Started by user Sumedh Basarkod [EnvInject] - Loading node environment variables. Building remotely on slv-s63702-24g-12c (P0 jython_slave IPv6 bkrs3) in workspace /data/workspace/rebalance_sumedh_temp [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/sumedhpb/TAF/ > /usr/bin/git init /data/workspace/rebalance_sumedh_temp # timeout=10 Fetching upstream changes from git://github.com/sumedhpb/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/sumedhpb/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/sumedhpb/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/sumedhpb/TAF/ # timeout=10 Fetching upstream changes from git://github.com/sumedhpb/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/sumedhpb/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 30633213d865ed79a162730dc116c781a8f391c0 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 30633213d865ed79a162730dc116c781a8f391c0 > /usr/bin/git rev-list 30633213d865ed79a162730dc116c781a8f391c0 # timeout=10 [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins306136414872825590.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:172.23.105.215 port:8091 [_2] ip:172.23.105.217 port:8091 [_3] ip:172.23.105.219 port:8091 [_4] ip:172.23.105.220 port:8091 [_5] ip:172.23.106.237 port:8091 [_6] ip:172.23.107.90 port:8091' + git clone https://github.com/gradle/guides.git Cloning into 'guides'... [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins6681342113940089206.sh + guides/gradlew --refresh-dependencies install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p version=7.0.0-4721,product=cb,parallel=True > Task :install 2021-03-19 01:25:22,177 - root - INFO - Done with uninstall on 172.23.106.237. 2021-03-19 01:25:22,891 - root - INFO - Done with uninstall on 172.23.105.219. 2021-03-19 01:25:25,947 - root - INFO - Done with uninstall on 172.23.107.90. 2021-03-19 01:25:27,111 - root - INFO - Done with uninstall on 172.23.105.217. 2021-03-19 01:25:29,441 - root - INFO - Done with uninstall on 172.23.105.220. 2021-03-19 01:25:29,723 - root - INFO - Done with uninstall on 172.23.105.215. 2021-03-19 01:29:53,226 - root - INFO - Done with install on 172.23.106.237. 2021-03-19 01:29:58,081 - root - INFO - Done with install on 172.23.105.217. 2021-03-19 01:29:59,191 - root - INFO - Done with install on 172.23.107.90. 2021-03-19 01:30:00,697 - root - INFO - Done with install on 172.23.105.220. 2021-03-19 01:30:02,903 - root - INFO - Done with install on 172.23.105.215. 2021-03-19 01:30:12,290 - root - INFO - Done with install on 172.23.105.219. 2021-03-19 01:30:54,960 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.237 2021-03-19 01:30:55,150 - root - INFO - Done with init on 172.23.106.237. 2021-03-19 01:30:55,176 - root - INFO - Done with cleanup on 172.23.106.237. 2021-03-19 01:30:59,457 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.217 2021-03-19 01:30:59,624 - root - INFO - Done with init on 172.23.105.217. 2021-03-19 01:30:59,655 - root - INFO - Done with cleanup on 172.23.105.217. 2021-03-19 01:31:00,588 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.107.90 2021-03-19 01:31:00,756 - root - INFO - Done with init on 172.23.107.90. 2021-03-19 01:31:00,785 - root - INFO - Done with cleanup on 172.23.107.90. 2021-03-19 01:31:02,107 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.220 2021-03-19 01:31:02,266 - root - INFO - Done with init on 172.23.105.220. 2021-03-19 01:31:02,295 - root - INFO - Done with cleanup on 172.23.105.220. 2021-03-19 01:31:04,334 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.215 2021-03-19 01:31:04,503 - root - INFO - Done with init on 172.23.105.215. 2021-03-19 01:31:04,532 - root - INFO - Done with cleanup on 172.23.105.215. 2021-03-19 01:31:13,657 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.219 2021-03-19 01:31:13,812 - root - INFO - Done with init on 172.23.105.219. 2021-03-19 01:31:13,834 - root - INFO - Done with cleanup on 172.23.105.219. 2021-03-19 01:31:30,913 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-19 01:31:30,934 - root - INFO - node:172.23.105.215:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-19 01:31:30,963 - root - INFO - node:172.23.105.217:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-19 01:31:30,986 - root - INFO - node:172.23.105.219:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-19 01:31:31,006 - root - INFO - node:172.23.105.220:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-19 01:31:31,026 - root - INFO - node:172.23.106.237:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-19 01:31:31,055 - root - INFO - node:172.23.107.90:8091 version:7.0.0-4721-enterprise aFamily:inet services:[u'kv'] 2021-03-19 01:31:31,055 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-19 01:31:31,055 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-19 01:31:31,056 - root - INFO - INSTALL COMPLETED ON: 172.23.105.215 2021-03-19 01:31:31,056 - root - INFO - INSTALL COMPLETED ON: 172.23.105.217 2021-03-19 01:31:31,056 - root - INFO - INSTALL COMPLETED ON: 172.23.105.219 2021-03-19 01:31:31,058 - root - INFO - INSTALL COMPLETED ON: 172.23.105.220 2021-03-19 01:31:31,058 - root - INFO - INSTALL COMPLETED ON: 172.23.106.237 2021-03-19 01:31:31,059 - root - INFO - INSTALL COMPLETED ON: 172.23.107.90 2021-03-19 01:31:31,059 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-19 01:31:31,062 - root - INFO - TOTAL INSTALL TIME = 411.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 6m 56s 1 actionable task: 1 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True -c conf/collections/collections_quorum_loss_failover.conf -m rest' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True -c conf/collections/collections_quorum_loss_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True -c conf/collections/collections_quorum_loss_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True -c conf/collections/collections_quorum_loss_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True -c conf/collections/collections_quorum_loss_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True -c conf/collections/collections_quorum_loss_failover.conf -m rest > Task :compileJava Note: /data/workspace/rebalance_sumedh_temp/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: conf/collections/collections_quorum_loss_failover.conf Prefix: bucket_collections.collections_quorum_loss.CollectionsQuorumLoss Global Test input params: {'cluster_name': 'durability_volume', 'conf_file': 'conf/collections/collections_quorum_loss_failover.conf', 'get-cbcollect-info': 'True', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 6, 'rerun': 'False', 'spec': 'collections_quorum_loss_failover'} Logs will be stored at /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Mar-19_01-31-37/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini rerun=False,get-cbcollect-info=True -t bucket_collections.collections_quorum_loss.CollectionsQuorumLoss.test_quorum_loss_failover,nodes_init=5,bucket_spec=multi_bucket.buckets_all_membase_for_quorum_loss,replicas=3,num_node_failures=3,failover_orchestrator=True,quota_percent=80,GROUP=P2' Test Input params: {'cluster_name': 'durability_volume', 'quota_percent': '80', 'ini': '/tmp/durability_volume.ini', 'get-cbcollect-info': 'True', 'replicas': '3', 'conf_file': 'conf/collections/collections_quorum_loss_failover.conf', 'spec': 'collections_quorum_loss_failover', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Mar-19_01-31-37/test_1', 'nodes_init': '5', 'GROUP': 'P2', 'failover_orchestrator': 'True', 'num_node_failures': '3', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_quorum_loss', 'case_number': 1} test_quorum_loss_failover (bucket_collections.collections_quorum_loss.CollectionsQuorumLoss) ... 2021-03-19 01:31:46,111 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsQuorumLoss setup started for test #1 test_quorum_loss_failover ========= 2021-03-19 01:31:47,051 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_quorum_loss_failover ========= 2021-03-19 01:31:48,219 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-19 01:34:28,859 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-19 01:34:48,400 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.105.215, 172.23.105.217, 172.23.105.219, 172.23.105.220, 172.23.106.237, 172.23.107.90 initialized 2021-03-19 01:34:48,421 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_quorum_loss_failover ========= 2021-03-19 01:34:48,454 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_quorum_loss_failover ========= 2021-03-19 01:35:03,640 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4721-enterprise | 3.56150543066 | Cluster node | | 172.23.105.217 | None | | | <--- IN --- | | 172.23.105.219 | None | | | <--- IN --- | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-19 01:35:03,667 | test | INFO | pool-1-thread-7 | [task:check:333] Rebalance - status: running, progress: 0.0 2021-03-19 01:35:13,746 | test | INFO | pool-1-thread-7 | [task:check:333] Rebalance - status: none, progress: 100 2021-03-19 01:35:13,776 | test | INFO | pool-1-thread-7 | [task:check:392] Rebalance completed with progress: 100% in 10.1380000114 sec 2021-03-19 01:35:14,088 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1796] Creating required buckets from template 2021-03-19 01:35:14,711 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2258] Waiting for all collections to be created 2021-03-19 01:35:14,997 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------------------------------------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------------------------------------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | YjHKmJQ6q%ozCYlOM_UcpJJOQdW-14-122000 | couchbase | 3 | none | 0 | 0 | 4194304000 | 0 | 0 | | ohbhR%GtYyS7tiuFGpm5jNN%i-UsmjuwaCNBebVOpEaDumJhpFHD4%nVfTHOzqWqLNJMI4mxhoTB3-14-213000 | couchbase | 3 | none | 0 | 0 | 0 | 0 | 0 | +-----------------------------------------------------------------------------------------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-19 01:35:15,000 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-19 01:35:39,576 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4654] Performing scope/collection specific operations 2021-03-19 01:35:39,595 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4744] Done Performing scope/collection specific operations 2021-03-19 01:35:49,713 | test | INFO | pool-1-thread-11 | [table_view:display:72] Ops trend for bucket 'ohbhR%GtYyS7tiuFGpm5jNN%i-UsmjuwaCNBebVOpEaDumJhpFHD4%nVfTHOzqWqLNJMI4mxhoTB3-14-213000' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................X | 3359.500 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-03-19 01:35:49,940 | test | INFO | pool-1-thread-12 | [table_view:display:72] Ops trend for bucket 'YjHKmJQ6q%ozCYlOM_UcpJJOQdW-14-122000' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 1104.300 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-03-19 01:35:50,433 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.105.215 | kv | 26.2165605096 | 4201840640 | 2977607680 | 53215232 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | | 172.23.106.237 | kv | 25.1589926227 | 4201840640 | 3035512832 | 51642368 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | | 172.23.105.220 | kv | 36.0420405024 | 4201840640 | 3032047616 | 35127296 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | | 172.23.105.217 | kv | 23.4167517875 | 4201844736 | 2996477952 | 6815744 / 3758092288 | 0 / 0 | 7.0.0-4721-enterprise | | 172.23.105.219 | kv | 34.9250698501 | 4201840640 | 3012665344 | 196726784 / 3758092288 | 23948 / 65543 | 7.0.0-4721-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-03-19 01:35:59,147 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4408] Validating collection stats and item counts 2021-03-19 01:36:19,335 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4290] Bucket UID mismatch. Expected: 0, Actual: 1 2021-03-19 01:36:19,336 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4425] Bucket manifest UID mismatch! 2021-03-19 01:36:53,696 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4290] Bucket UID mismatch. Expected: 0, Actual: 1 2021-03-19 01:36:53,697 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4425] Bucket manifest UID mismatch! 2021-03-19 01:37:19,701 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------------------------------------------------------------------------------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------------------------------------------------------------------------------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | YjHKmJQ6q%ozCYlOM_UcpJJOQdW-14-122000 | couchbase | 3 | none | 0 | 175000 | 4194304000 | 694550392 | 815207204 | | ohbhR%GtYyS7tiuFGpm5jNN%i-UsmjuwaCNBebVOpEaDumJhpFHD4%nVfTHOzqWqLNJMI4mxhoTB3-14-213000 | couchbase | 3 | none | 0 | 175000 | 4194304000 | 557652904 | 827401589 | +-----------------------------------------------------------------------------------------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-19 01:37:19,703 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_quorum_loss_failover ========= 2021-03-19 01:37:19,703 | test | INFO | MainThread | [collections_quorum_loss:servers_to_fail:157] changing master to ip:172.23.105.220 port:8091 ssh_username:root 2021-03-19 01:37:19,704 | test | INFO | MainThread | [collections_quorum_loss:servers_to_fail:164] also modifying self.nodes.in.cluster to [ip:172.23.106.237 port:8091 ssh_username:root, ip:172.23.105.217 port:8091 ssh_username:root, ip:172.23.105.219 port:8091 ssh_username:root, ip:172.23.105.220 port:8091 ssh_username:root, ip:172.23.105.215 port:8091 ssh_username:root] 2021-03-19 01:37:19,776 | test | INFO | MainThread | [collections_quorum_loss:test_quorum_loss_failover:272] Failing over nodes explicitly [ip:172.23.105.215 port:8091 ssh_username:root, ip:172.23.105.217 port:8091 ssh_username:root, ip:172.23.105.219 port:8091 ssh_username:root] 2021-03-19 01:37:20,200 | test | ERROR | pool-1-thread-17 | [rest_client:_http_request:748] POST http://172.23.106.237:8091/controller/failOver body: otpNode=ns_1%40172.23.105.215&otpNode=ns_1%40172.23.105.217&otpNode=ns_1%40172.23.105.219&allowUnsafe=true headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: status: 500, content: Unexpected server error: {error, {not_in_peers,'ns_1@172.23.105.215', ['ns_1@172.23.105.220', 'ns_1@172.23.106.237']}} Unexpected server error: {error, {not_in_peers,'ns_1@172.23.105.215', ['ns_1@172.23.105.220', 'ns_1@172.23.106.237']}} auth: Administrator:password 2021-03-19 01:37:20,203 | test | ERROR | pool-1-thread-17 | [rest_client:fail_over:1291] [u'ns_1@172.23.105.215', u'ns_1@172.23.105.217', u'ns_1@172.23.105.219'] - Failover error: Unexpected server error: {error, {not_in_peers,'ns_1@172.23.105.215', ['ns_1@172.23.105.220', 'ns_1@172.23.106.237']}} ERROR 2021-03-19 01:37:21,226 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: YjHKmJQ6q%ozCYlOM_UcpJJOQdW-14-122000, Active Resident ratio(DGM): 100% 2021-03-19 01:37:21,226 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: YjHKmJQ6q%ozCYlOM_UcpJJOQdW-14-122000, Replica Resident ratio(DGM): 100% Build was aborted Aborted by Sumedh Basarkod Archiving artifacts Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: ABORTED