Started by user Sumedh Basarkod [EnvInject] - Loading node environment variables. Building remotely on slv-s63802-24g-12c (P0 jython_slave) in workspace /data/workspace/rebalance_sumedh_temp [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/rebalance_sumedh_temp # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 6f3bf797cd97fd4a22293ed3c85de305fe382d9d (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 6f3bf797cd97fd4a22293ed3c85de305fe382d9d > /usr/bin/git rev-list 23e3b5983509c3db4630a4998ad233937d5b5fd3 # timeout=10 [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins7447398465545233688.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:172.23.105.215 port:8091 [_2] ip:172.23.105.217 port:8091 [_3] ip:172.23.105.219 port:8091 [_4] ip:172.23.105.220 port:8091 [_5] ip:172.23.106.237 port:8091 [_6] ip:172.23.107.90 port:8091' + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + git fetch http://review.couchbase.org/TAF refs/changes/04/148904/6 From http://review.couchbase.org/TAF * branch refs/changes/04/148904/6 -> FETCH_HEAD + git cherry-pick FETCH_HEAD [detached HEAD b601324] CBQE-6416: Do not check for number of failovered nodes Author: sumedhpb 2 files changed, 39 insertions(+), 21 deletions(-) [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins3252375373648847636.sh + guides/gradlew --refresh-dependencies install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True' Starting a Gradle Daemon, 37 busy and 1 incompatible and 1 stopped Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True > Task :install 2021-03-18 05:43:58,473 - root - INFO - Done with uninstall on 172.23.107.90. 2021-03-18 05:43:58,483 - root - INFO - Done with uninstall on 172.23.106.237. 2021-03-18 05:43:59,085 - root - INFO - Done with uninstall on 172.23.105.219. 2021-03-18 05:43:59,153 - root - INFO - Done with uninstall on 172.23.105.217. 2021-03-18 05:43:59,311 - root - INFO - Done with uninstall on 172.23.105.220. 2021-03-18 05:43:59,698 - root - INFO - Done with uninstall on 172.23.105.215. 2021-03-18 05:48:28,273 - root - INFO - Done with install on 172.23.105.215. 2021-03-18 05:48:30,256 - root - INFO - Done with install on 172.23.105.220. 2021-03-18 05:48:41,668 - root - INFO - Done with install on 172.23.107.90. 2021-03-18 05:48:41,694 - root - INFO - Done with install on 172.23.105.217. 2021-03-18 05:48:42,572 - root - INFO - Done with install on 172.23.106.237. 2021-03-18 05:48:44,559 - root - INFO - Done with install on 172.23.105.219. 2021-03-18 05:49:29,990 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.215 2021-03-18 05:49:30,178 - root - INFO - Done with init on 172.23.105.215. 2021-03-18 05:49:30,219 - root - INFO - Done with cleanup on 172.23.105.215. 2021-03-18 05:49:31,631 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.220 2021-03-18 05:49:31,799 - root - INFO - Done with init on 172.23.105.220. 2021-03-18 05:49:31,819 - root - INFO - Done with cleanup on 172.23.105.220. 2021-03-18 05:49:43,040 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.107.90 2021-03-18 05:49:43,072 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.217 2021-03-18 05:49:43,207 - root - INFO - Done with init on 172.23.107.90. 2021-03-18 05:49:43,230 - root - INFO - Done with init on 172.23.105.217. 2021-03-18 05:49:43,233 - root - INFO - Done with cleanup on 172.23.107.90. 2021-03-18 05:49:43,255 - root - INFO - Done with cleanup on 172.23.105.217. 2021-03-18 05:49:43,937 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.237 2021-03-18 05:49:44,128 - root - INFO - Done with init on 172.23.106.237. 2021-03-18 05:49:44,160 - root - INFO - Done with cleanup on 172.23.106.237. 2021-03-18 05:49:45,924 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.219 2021-03-18 05:49:46,078 - root - INFO - Done with init on 172.23.105.219. 2021-03-18 05:49:46,111 - root - INFO - Done with cleanup on 172.23.105.219. 2021-03-18 05:50:03,525 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-18 05:50:03,548 - root - INFO - node:172.23.105.215:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-18 05:50:03,568 - root - INFO - node:172.23.105.217:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-18 05:50:03,588 - root - INFO - node:172.23.105.219:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-18 05:50:03,607 - root - INFO - node:172.23.105.220:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-18 05:50:03,628 - root - INFO - node:172.23.106.237:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-18 05:50:03,648 - root - INFO - node:172.23.107.90:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-18 05:50:03,648 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-18 05:50:03,648 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-18 05:50:03,648 - root - INFO - INSTALL COMPLETED ON: 172.23.105.215 2021-03-18 05:50:03,650 - root - INFO - INSTALL COMPLETED ON: 172.23.105.217 2021-03-18 05:50:03,650 - root - INFO - INSTALL COMPLETED ON: 172.23.105.219 2021-03-18 05:50:03,650 - root - INFO - INSTALL COMPLETED ON: 172.23.105.220 2021-03-18 05:50:03,651 - root - INFO - INSTALL COMPLETED ON: 172.23.106.237 2021-03-18 05:50:03,651 - root - INFO - INSTALL COMPLETED ON: 172.23.107.90 2021-03-18 05:50:03,651 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-18 05:50:03,654 - root - INFO - TOTAL INSTALL TIME = 411.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 7m 1s 1 actionable task: 1 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p rerun=False -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,failover_action=firewall,num_node_failures=3,quota_percent=80,GROUP=P1 -m rest' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p rerun=False -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,failover_action=firewall,num_node_failures=3,quota_percent=80,GROUP=P1 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p rerun=False -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,failover_action=firewall,num_node_failures=3,quota_percent=80,GROUP=P1 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p rerun=False -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,failover_action=firewall,num_node_failures=3,quota_percent=80,GROUP=P1 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p rerun=False -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,failover_action=firewall,num_node_failures=3,quota_percent=80,GROUP=P1 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p rerun=False -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,failover_action=firewall,num_node_failures=3,quota_percent=80,GROUP=P1 -m rest > Task :compileJava Note: /data/workspace/rebalance_sumedh_temp/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Global Test input params: {'cluster_name': 'durability_volume', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 6, 'rerun': 'False'} Logs will be stored at /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Mar-18_05-50-11/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini rerun=False -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,failover_action=firewall,num_node_failures=3,quota_percent=80,GROUP=P1' Test Input params: {'cluster_name': 'durability_volume', 'quota_percent': '80', 'ini': '/tmp/durability_volume.ini', 'replicas': '3', 'failover_action': 'firewall', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Mar-18_05-50-11/test_1', 'nodes_init': '5', 'GROUP': 'P1', 'num_node_failures': '3', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_quorum_loss', 'case_number': 1} test_quorum_loss_failover (bucket_collections.collections_quorum_loss.CollectionsQuorumLoss) ... 2021-03-18 05:50:19,740 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsQuorumLoss setup started for test #1 test_quorum_loss_failover ========= 2021-03-18 05:50:20,617 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_quorum_loss_failover ========= 2021-03-18 05:50:21,625 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-18 05:53:02,658 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-18 05:53:21,838 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.105.215, 172.23.105.217, 172.23.105.219, 172.23.105.220, 172.23.106.237, 172.23.107.90 initialized 2021-03-18 05:53:21,858 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_quorum_loss_failover ========= 2021-03-18 05:53:21,890 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_quorum_loss_failover ========= 2021-03-18 05:53:37,832 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4721-enterprise | 4.46293494705 | Cluster node | | 172.23.105.217 | None | | | <--- IN --- | | 172.23.105.219 | None | | | <--- IN --- | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-18 05:53:37,857 | test | INFO | pool-1-thread-7 | [task:check:333] Rebalance - status: running, progress: 0.0 2021-03-18 05:53:47,921 | test | INFO | pool-1-thread-7 | [task:check:333] Rebalance - status: none, progress: 100 2021-03-18 05:53:47,947 | test | INFO | pool-1-thread-7 | [task:check:392] Rebalance completed with progress: 100% in 10.1159999371 sec 2021-03-18 05:53:48,154 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1796] Creating required buckets from template 2021-03-18 05:53:48,803 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2258] Waiting for all collections to be created 2021-03-18 05:53:49,038 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | A9%1Zc1YY4wOO3%lD-48-277000 | couchbase | 3 | none | 0 | 0 | 4194304000 | 0 | 0 | | DMOCEALabmUOiFZj6L_ccczdP-48-189000 | couchbase | 3 | none | 0 | 0 | 0 | 0 | 0 | +-------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-18 05:53:49,039 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-18 05:54:14,137 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4654] Performing scope/collection specific operations 2021-03-18 05:54:14,157 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4744] Done Performing scope/collection specific operations 2021-03-18 05:54:21,424 | test | INFO | pool-1-thread-11 | [table_view:display:72] Ops trend for bucket 'DMOCEALabmUOiFZj6L_ccczdP-48-189000' +-------+---------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................X | 434.400 | +-------+---------------------------------------------------------------------------------------------------------+---------+ 2021-03-18 05:54:21,602 | test | INFO | pool-1-thread-12 | [table_view:display:72] Ops trend for bucket 'A9%1Zc1YY4wOO3%lD-48-277000' +-------+---------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................X | 363.300 | +-------+---------------------------------------------------------------------------------------------------------+---------+ 2021-03-18 05:54:21,885 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.105.215 | kv | 12.7263453201 | 4201840640 | 3073335296 | 53215232 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | | 172.23.106.237 | kv | 14.1794871795 | 4201840640 | 3110219776 | 51642368 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | | 172.23.105.220 | kv | 34.2835327055 | 4201840640 | 3093590016 | 35127296 / 3758092288 | 7963 / 22577 | 7.0.0-4721-enterprise | | 172.23.105.217 | kv | 15.6010230179 | 4201844736 | 3085824000 | 6815744 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | | 172.23.105.219 | kv | 10.7499360123 | 4201840640 | 3071295488 | 196837376 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-03-18 05:54:30,480 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4408] Validating collection stats and item counts 2021-03-18 05:54:51,562 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4290] Bucket UID mismatch. Expected: 0, Actual: 1 2021-03-18 05:54:51,563 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4425] Bucket manifest UID mismatch! 2021-03-18 05:55:27,019 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4290] Bucket UID mismatch. Expected: 0, Actual: 1 2021-03-18 05:55:27,020 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4425] Bucket manifest UID mismatch! 2021-03-18 05:55:56,053 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------------------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------------------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | A9%1Zc1YY4wOO3%lD-48-277000 | couchbase | 3 | none | 0 | 175000 | 4194304000 | 576985160 | 844152192 | | DMOCEALabmUOiFZj6L_ccczdP-48-189000 | couchbase | 3 | none | 0 | 175000 | 4194304000 | 637234472 | 802156547 | +-------------------------------------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-18 05:55:56,055 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_quorum_loss_failover ========= 2021-03-18 05:55:56,092 | test | INFO | MainThread | [collections_quorum_loss:test_quorum_loss_failover:261] Inducing failure firewall on nodes: [ip:172.23.105.217 port:8091 ssh_username:root, ip:172.23.105.219 port:8091 ssh_username:root, ip:172.23.105.220 port:8091 ssh_username:root] 2021-03-18 05:55:58,548 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Wait before failing over 2021-03-18 05:56:58,549 | test | INFO | MainThread | [collections_quorum_loss:test_quorum_loss_failover:266] Failing over nodes explicitly [ip:172.23.105.217 port:8091 ssh_username:root, ip:172.23.105.219 port:8091 ssh_username:root, ip:172.23.105.220 port:8091 ssh_username:root] 2021-03-18 05:57:15,174 | test | ERROR | pool-1-thread-14 | [rest_client:_http_request:748] POST http://172.23.105.215:8091/controller/failOver body: otpNode=ns_1%40172.23.105.217&otpNode=ns_1%40172.23.105.219&otpNode=ns_1%40172.23.105.220&allowUnsafe=true headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password 2021-03-18 05:57:15,177 | test | ERROR | pool-1-thread-14 | [rest_client:fail_over:1291] [u'ns_1@172.23.105.217', u'ns_1@172.23.105.219', u'ns_1@172.23.105.220'] - Failover error: ["Unexpected server error, request logged."] ERROR 2021-03-18 05:57:18,424 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: A9%1Zc1YY4wOO3%lD-48-277000, Active Resident ratio(DGM): 100% 2021-03-18 05:57:18,424 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: A9%1Zc1YY4wOO3%lD-48-277000, Replica Resident ratio(DGM): 100% 2021-03-18 05:57:56,941 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: DMOCEALabmUOiFZj6L_ccczdP-48-189000, Active Resident ratio(DGM): 100% 2021-03-18 05:57:56,943 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: DMOCEALabmUOiFZj6L_ccczdP-48-189000, Replica Resident ratio(DGM): 100% 2021-03-18 05:58:51,095 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:487] Skipping get_trace !! 2021-03-18 05:59:21,444 | infra | ERROR | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.105.215:8091/pools/default/buckets/A9%251Zc1YY4wOO3%25lD-48-277000 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-03-18 05:59:22,269 | infra | ERROR | MainThread | [remote_util:log_command_output:3064] wc: /opt/couchbase/var/lib/couchbase/data: Is a directory 2021-03-18 05:59:22,270 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1507] Unable to get timings for bucket: global name 'StatsCommon' is not defined Build was aborted Aborted by Sumedh Basarkod Archiving artifacts Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: ABORTED