Started by user Sumedh Basarkod [EnvInject] - Loading node environment variables. Building remotely on slv-s63702-24g-12c (P0 jython_slave IPv6 bkrs3) in workspace /data/workspace/rebalance_sumedh_temp [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/rebalance_sumedh_temp # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 829cfa861bf2eed092382b9d006f801a3ea8f388 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 829cfa861bf2eed092382b9d006f801a3ea8f388 > /usr/bin/git rev-list 829cfa861bf2eed092382b9d006f801a3ea8f388 # timeout=10 [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins2583546467068478528.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 [_1] ip:172.23.105.215 port:8091 [_2] ip:172.23.105.217 port:8091 [_3] ip:172.23.105.219 port:8091 [_4] ip:172.23.106.237 port:8091 [_5] ip:172.23.107.90 port:8091' + git clone https://github.com/gradle/guides.git Cloning into 'guides'... [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins659260815114219343.sh + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini GROUP=set2,rerun=False,get-cbcollect-info=True -t bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split,nodes_init=4,bucket_spec=single_bucket.buckets_all_membase_for_rebalance_tests_more_collections,allow_unsafe=True,majority=second_half,GROUP=set2 -m rest' Starting a Gradle Daemon, 22 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini GROUP=set2,rerun=False,get-cbcollect-info=True -t bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split,nodes_init=4,bucket_spec=single_bucket.buckets_all_membase_for_rebalance_tests_more_collections,allow_unsafe=True,majority=second_half,GROUP=set2 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini GROUP=set2,rerun=False,get-cbcollect-info=True -t bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split,nodes_init=4,bucket_spec=single_bucket.buckets_all_membase_for_rebalance_tests_more_collections,allow_unsafe=True,majority=second_half,GROUP=set2 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini GROUP=set2,rerun=False,get-cbcollect-info=True -t bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split,nodes_init=4,bucket_spec=single_bucket.buckets_all_membase_for_rebalance_tests_more_collections,allow_unsafe=True,majority=second_half,GROUP=set2 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini GROUP=set2,rerun=False,get-cbcollect-info=True -t bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split,nodes_init=4,bucket_spec=single_bucket.buckets_all_membase_for_rebalance_tests_more_collections,allow_unsafe=True,majority=second_half,GROUP=set2 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini GROUP=set2,rerun=False,get-cbcollect-info=True -t bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split,nodes_init=4,bucket_spec=single_bucket.buckets_all_membase_for_rebalance_tests_more_collections,allow_unsafe=True,majority=second_half,GROUP=set2 -m rest > Task :compileJava Note: /data/workspace/rebalance_sumedh_temp/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Global Test input params: {'cluster_name': 'durability_volume', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 5} Logs will be stored at /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-May-10_02-33-36/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -t bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split,nodes_init=4,bucket_spec=single_bucket.buckets_all_membase_for_rebalance_tests_more_collections,allow_unsafe=True,majority=second_half,GROUP=set2' Test Input params: {'cluster_name': 'durability_volume', 'allow_unsafe': 'True', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 5, 'logs_folder': '/data/workspace/rebalance_sumedh_temp/logs/testrunner-21-May-10_02-33-36/test_1', 'rerun': False, 'nodes_init': '4', 'GROUP': 'set2', 'bucket_spec': 'single_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'majority': 'second_half', 'case_number': 1} test_quorum_loss_with_network_split (bucket_collections.collections_network_split.CollectionsNetworkSplit) ... 2021-05-10 02:33:43,641 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsNetworkSplit setup started for test #1 test_quorum_loss_with_network_split ========= 2021-05-10 02:33:44,578 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #1 test_quorum_loss_with_network_split ========= 2021-05-10 02:33:45,588 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.105.215:8091 during connect (<_realsocket at 0x6 type=client open_count=1 channel=[id: 0x3dc1c59f, 0.0.0.0/0.0.0.0:55342] timeout=300.0>) 2021-05-10 02:33:50,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.215:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.215:8091 during connect (<_realsocket at 0x7 type=client open_count=1 channel=[id: 0xa1023972, 0.0.0.0/0.0.0.0:36746] timeout=300.0>) 2021-05-10 02:33:52,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.215:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.215:8091 during connect (<_realsocket at 0x8 type=client open_count=1 channel=[id: 0xa77ac0ba, 0.0.0.0/0.0.0.0:60370] timeout=300.0>) 2021-05-10 02:33:54,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.215:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.215:8091 during connect (<_realsocket at 0x9 type=client open_count=1 channel=[id: 0x899a13dd, 0.0.0.0/0.0.0.0:38148] timeout=300.0>) 2021-05-10 02:33:56,877 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.215:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.217:8091 during connect (<_realsocket at 0xa type=client open_count=1 channel=[id: 0x8b86b00b, 0.0.0.0/0.0.0.0:35156] timeout=300.0>) 2021-05-10 02:34:03,571 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.217:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.217:8091 during connect (<_realsocket at 0xb type=client open_count=1 channel=[id: 0x0dc50721, 0.0.0.0/0.0.0.0:60618] timeout=300.0>) 2021-05-10 02:34:05,588 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.217:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.217:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0xbcef58c6, 0.0.0.0/0.0.0.0:45472] timeout=300.0>) 2021-05-10 02:34:07,605 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.217:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.217:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xfdede815, 0.0.0.0/0.0.0.0:45513] timeout=300.0>) 2021-05-10 02:34:09,622 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.217:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.217:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x7250074e, 0.0.0.0/0.0.0.0:45492] timeout=300.0>) 2021-05-10 02:34:11,641 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.217:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.217:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x9fc917a5, 0.0.0.0/0.0.0.0:35730] timeout=300.0>) 2021-05-10 02:34:13,657 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.217:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.217:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x9a7fc088, 0.0.0.0/0.0.0.0:53765] timeout=300.0>) 2021-05-10 02:34:15,673 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.217:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x9ebd6920, 0.0.0.0/0.0.0.0:43189] timeout=300.0>) 2021-05-10 02:34:22,331 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xb70b5a27, 0.0.0.0/0.0.0.0:53624] timeout=300.0>) 2021-05-10 02:34:24,345 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x4cd5b9c9, 0.0.0.0/0.0.0.0:37898] timeout=300.0>) 2021-05-10 02:34:26,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x8bb3b255, 0.0.0.0/0.0.0.0:34456] timeout=300.0>) 2021-05-10 02:34:28,529 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x18eb69ea, 0.0.0.0/0.0.0.0:46075] timeout=300.0>) 2021-05-10 02:34:30,543 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x2bb5daf3, 0.0.0.0/0.0.0.0:48520] timeout=300.0>) 2021-05-10 02:34:32,556 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x6e13a1f6, 0.0.0.0/0.0.0.0:38853] timeout=300.0>) 2021-05-10 02:34:34,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.237:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0xd01bcd0b, 0.0.0.0/0.0.0.0:34091] timeout=300.0>) 2021-05-10 02:34:41,325 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.237:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.237:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x9337fcc7, 0.0.0.0/0.0.0.0:60025] timeout=300.0>) 2021-05-10 02:34:43,338 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.237:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.237:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xd07ea9b8, 0.0.0.0/0.0.0.0:55895] timeout=300.0>) 2021-05-10 02:34:45,354 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.237:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.237:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0xd43e3342, 0.0.0.0/0.0.0.0:46893] timeout=300.0>) 2021-05-10 02:34:47,368 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.237:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.237:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0xa4a65915, 0.0.0.0/0.0.0.0:35535] timeout=300.0>) 2021-05-10 02:34:49,381 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.237:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.237:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0xaac1b8d4, 0.0.0.0/0.0.0.0:40800] timeout=300.0>) 2021-05-10 02:34:51,394 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.237:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.237:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xe6bf12f3, 0.0.0.0/0.0.0.0:57919] timeout=300.0>) 2021-05-10 02:34:53,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.237:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.90:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x052ae2d3, 0.0.0.0/0.0.0.0:46704] timeout=300.0>) 2021-05-10 02:35:00,756 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.90:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.90:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0xe5790b2a, 0.0.0.0/0.0.0.0:54607] timeout=300.0>) 2021-05-10 02:35:02,770 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.90:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.90:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0xefc14e16, 0.0.0.0/0.0.0.0:51804] timeout=300.0>) 2021-05-10 02:35:04,785 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.90:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.90:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0xe136f4b3, 0.0.0.0/0.0.0.0:55123] timeout=300.0>) 2021-05-10 02:35:06,801 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.90:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 02:35:23,010 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-10 02:35:36,993 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.105.215, 172.23.105.217, 172.23.105.219, 172.23.106.237, 172.23.107.90 initialized 2021-05-10 02:35:37,009 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #1 test_quorum_loss_with_network_split ========= 2021-05-10 02:35:37,032 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #1 test_quorum_loss_with_network_split ========= 2021-05-10 02:35:53,947 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.105.215 | kv | 7.0.0-5127-enterprise | 3.4188034188 | Cluster node | | 172.23.105.217 | None | | | <--- IN --- | | 172.23.105.219 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-10 02:35:53,966 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-10 02:36:04,022 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-10 02:36:04,046 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.0980000496 sec 2021-05-10 02:36:04,250 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-10 02:36:04,525 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-10 02:36:04,617 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | default | couchbase | 3 | none | 0 | 0 | 8388608000 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-05-10 02:36:04,618 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-10 02:36:21,605 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-10 02:36:21,621 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-10 02:36:29,842 | test | INFO | pool-2-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 3929.549 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-10 02:36:29,937 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.105.215 | kv | 17.6635276152 | 4201840640 | 3426041856 | 254541824 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.106.237 | kv | 17.6891615542 | 4201840640 | 3448643584 | 116129792 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.105.217 | kv | 17.7099236641 | 4201844736 | 3460968448 | 275558400 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.105.219 | kv | 28.6077882413 | 4201840640 | 3443486720 | 279261184 / 3758092288 | 39229 / 78826 | 7.0.0-5127-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-05-10 02:36:38,834 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-10 02:36:51,142 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4287] Bucket UID mismatch. Expected: 0, Actual: 1 2021-05-10 02:36:51,144 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4422] Bucket manifest UID mismatch! 2021-05-10 02:36:59,779 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 833984064 | 556314064 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-10 02:36:59,780 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #1 test_quorum_loss_with_network_split ========= 2021-05-10 02:36:59,782 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-10 02:37:07,934 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-10 02:37:09,219 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.105.219 in 172.23.105.215 2021-05-10 02:37:09,829 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.105.215 in 172.23.105.219 2021-05-10 02:37:10,454 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.106.237 in 172.23.105.215 2021-05-10 02:37:11,066 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.105.215 in 172.23.106.237 2021-05-10 02:37:11,706 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.105.219 in 172.23.105.217 2021-05-10 02:37:12,398 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.105.217 in 172.23.105.219 2021-05-10 02:37:13,032 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.106.237 in 172.23.105.217 2021-05-10 02:37:13,828 | test | INFO | MainThread | [collections_network_split:block_traffic_between_two_nodes:56] Blocking traffic from 172.23.105.217 in 172.23.106.237 2021-05-10 02:37:13,864 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Wait for network split to finish 2021-05-10 02:38:13,865 | test | INFO | MainThread | [collections_network_split:test_quorum_loss_with_network_split:270] First half nodes [ip:172.23.105.215 port:8091 ssh_username:root, ip:172.23.105.217 port:8091 ssh_username:root] 2021-05-10 02:38:13,865 | test | INFO | MainThread | [collections_network_split:test_quorum_loss_with_network_split:271] Second half nodes [ip:172.23.105.219 port:8091 ssh_username:root, ip:172.23.106.237 port:8091 ssh_username:root] 2021-05-10 02:38:13,865 | test | INFO | MainThread | [collections_network_split:test_quorum_loss_with_network_split:272] Failing over nodes: [ip:172.23.105.219 port:8091 ssh_username:root, ip:172.23.106.237 port:8091 ssh_username:root] 2021-05-10 02:38:27,967 | test | ERROR | pool-2-thread-29 | [rest_client:_http_request:748] POST http://172.23.105.215:8091/controller/failOver body: otpNode=ns_1%40172.23.105.219&otpNode=ns_1%40172.23.106.237&allowUnsafe=true headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: status: 500, content: Unexpected server error: no_leader Unexpected server error: no_leader auth: Administrator:password 2021-05-10 02:38:27,969 | test | ERROR | pool-2-thread-29 | [rest_client:fail_over:1276] [u'ns_1@172.23.105.219', u'ns_1@172.23.106.237'] - Failover error: Unexpected server error: no_leader ERROR 2021-05-10 02:38:41,161 | test | INFO | MainThread | [collections_network_split:wipe_config_on_removed_nodes:115] Wiping node config and restarting server on ip:172.23.105.219 port:8091 ssh_username:root Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x17935 type=client open_count=1 channel=[id: 0x7d11336e, 0.0.0.0/0.0.0.0:58635] timeout=5.0>) 2021-05-10 02:39:05,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.219:8091 during connect (<_realsocket at 0x17936 type=client open_count=1 channel=[id: 0x75fa3293, 0.0.0.0/0.0.0.0:51873] timeout=5.0>) 2021-05-10 02:39:07,283 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.219:8091/nodes/self error [Errno 111] Connection refused 2021-05-10 02:39:09,502 | test | INFO | MainThread | [collections_network_split:wipe_config_on_removed_nodes:115] Wiping node config and restarting server on ip:172.23.106.237 port:8091 ssh_username:root 2021-05-10 02:39:37,773 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-10 02:39:37,775 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-10 02:39:52,901 | infra | ERROR | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.105.215:8091/pools/default/buckets/default/scopes/_default/collections/v_-zMjFmuKB6sldbNKQ-4-288000 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password 2021-05-10 02:39:52,904 | test | ERROR | MainThread | [bucket_ready_functions:drop_collection:893] Collection 'default:_default:v_-zMjFmuKB6sldbNKQ-4-288000' delete failed: ["Unexpected server error, request logged."] ERROR ====================================================================== ERROR: test_quorum_loss_with_network_split (bucket_collections.collections_network_split.CollectionsNetworkSplit) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_network_split.py", line 273, in test_quorum_loss_with_network_split result = self.task.failover(otp_nodes, failover_nodes=self.nodes_failover, File "lib/couchbase_helper/cluster.py", line 929, in failover self.jython_task_manager.get_task_result(_task) File "lib/Jython_tasks/task_manager.py", line 28, in get_task_result result = future.get() ExecutionException: java.util.concurrent.ExecutionException: Traceback (most recent call last): File "lib/Jython_tasks/task.py", line 5841, in call self.set_exception(e) File "lib/Jython_tasks/task.py", line 92, in set_exception raise BaseException(self.exception) BaseException: Failover Node failed: Unexpected server error: no_leader ====================================================================== ERROR: test_quorum_loss_with_network_split (bucket_collections.collections_network_split.CollectionsNetworkSplit) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_network_split.py", line 36, in tearDown super(CollectionsNetworkSplit, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 107, in tearDown self.bucket_util.remove_scope_collections_for_bucket(bucket) File "couchbase_utils/bucket_utils/bucket_ready_functions.py", line 4436, in remove_scope_collections_for_bucket self.drop_collection(self.cluster.master, File "couchbase_utils/bucket_utils/bucket_ready_functions.py", line 896, in drop_collection raise Exception("delete_collection") Exception: delete_collection ---------------------------------------------------------------------- During the test, Remote Connections: 52, Disconnections: 51 SDK Connections: 30, Disconnections: 0 !!!!!! CRITICAL :: Shell disconnection mismatch !!!!! !!!!!! CRITICAL :: SDK disconnection mismatch !!!!! Ran 1 test in 374.057s FAILED (errors=2) summary so far suite bucket_collections.collections_network_split.CollectionsNetworkSplit , pass 0 , fail 1 failures so far... bucket_collections.collections_network_split.CollectionsNetworkSplit.test_quorum_loss_with_network_split testrunner logs, diags and results are available under /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-May-10_02-33-36/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 6m 31s 2 actionable tasks: 2 executed Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Could not determine description. [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE