Started by user Sumedh Basarkod [EnvInject] - Loading node environment variables. Building remotely on slv-sc2402-32g-12c (volume_test_collections) in workspace /data/workspace/temp_durability_volume [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_durability_volume # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 1a1810c92cdc1414b849f82c0059083fce88d845 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 1a1810c92cdc1414b849f82c0059083fce88d845 > /usr/bin/git rev-list ea365e8e36ce5a34f1b088fac43468c9fdd9eff5 # timeout=10 [temp_durability_volume] $ /bin/sh -xe /tmp/jenkins3697510821473152993.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 data_path:/data/ [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 [_1] ip:172.23.105.175 [_2] ip:172.23.106.233 [_3] ip:172.23.106.236 [_4] ip:172.23.106.238 [_5] ip:172.23.106.250 [_6] ip:172.23.106.251 [_7] ip:172.23.121.74 [_8] ip:172.23.121.78' + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 data_path:/data/ [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 9:_9 10:_10 11:_11 12:_12 13:_13 14:_14 15:_15 16:_16 17:_17 18:_18 19:_19 20:_20 [_1] ip:172.23.105.175 [_2] ip:172.23.106.233 [_3] ip:172.23.106.236 [_4] ip:172.23.106.238 [_5] ip:172.23.106.250 [_6] ip:172.23.106.251 [_7] ip:172.23.121.74 [_8] ip:172.23.121.78 [_9] ip:172.23.107.43 [_10] ip:172.23.107.58 [_11] ip:172.23.107.44 [_12] ip:172.23.107.45 [_13] ip:172.23.107.54 [_14] ip:172.23.107.47 [_15] ip:172.23.107.78 [_16] ip:172.23.107.84 [_17] ip:172.23.107.85 [_18] ip:172.23.107.88 [_19] ip:172.23.107.91 [_20] ip:172.23.107.107' + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + cp guides/gradle.properties . + git fetch http://review.couchbase.org/TAF refs/changes/03/152303/9 From http://review.couchbase.org/TAF * branch refs/changes/03/152303/9 -> FETCH_HEAD + git cherry-pick FETCH_HEAD [detached HEAD 6782b79] Create indexes in incremental fashion in volume metadata test Author: sumedhpb 1 file changed, 57 insertions(+), 17 deletions(-) [temp_durability_volume] $ /bin/sh -xe /tmp/jenkins236964972007261123.sh + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + guides/gradlew --refresh-dependencies install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/20_node.ini -p version=7.0.0-5127,product=cb,parallel=True' Starting a Gradle Daemon, 72 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/20_node.ini -p version=7.0.0-5127,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/20_node.ini -p version=7.0.0-5127,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/20_node.ini -p version=7.0.0-5127,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/20_node.ini -p version=7.0.0-5127,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/20_node.ini -p version=7.0.0-5127,product=cb,parallel=True > Task :install ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2 2021-05-11 01:23:45,319 - root - INFO - Done with uninstall on 172.23.107.107. 2021-05-11 01:23:45,387 - root - INFO - Done with uninstall on 172.23.107.91. 2021-05-11 01:23:45,467 - root - INFO - Done with uninstall on 172.23.107.85. 2021-05-11 01:23:45,473 - root - INFO - Done with uninstall on 172.23.107.78. 2021-05-11 01:23:45,480 - root - INFO - Done with uninstall on 172.23.107.54. 2021-05-11 01:23:45,486 - root - INFO - Done with uninstall on 172.23.107.45. 2021-05-11 01:23:45,549 - root - INFO - Done with uninstall on 172.23.107.84. 2021-05-11 01:23:45,561 - root - INFO - Done with uninstall on 172.23.107.47. 2021-05-11 01:23:45,565 - root - INFO - Done with uninstall on 172.23.107.88. 2021-05-11 01:23:45,599 - root - INFO - Done with uninstall on 172.23.107.58. 2021-05-11 01:23:45,625 - root - INFO - Done with uninstall on 172.23.107.44. 2021-05-11 01:23:45,644 - root - INFO - Done with uninstall on 172.23.107.43. 2021-05-11 01:23:48,244 - root - INFO - Done with uninstall on 172.23.106.233. 2021-05-11 01:23:48,709 - root - INFO - Done with uninstall on 172.23.106.250. 2021-05-11 01:23:48,890 - root - INFO - Done with uninstall on 172.23.121.78. 2021-05-11 01:23:49,079 - root - INFO - Done with uninstall on 172.23.106.236. 2021-05-11 01:23:49,085 - root - INFO - Done with uninstall on 172.23.106.251. 2021-05-11 01:23:49,131 - root - INFO - Done with uninstall on 172.23.106.238. 2021-05-11 01:23:49,321 - root - INFO - Done with uninstall on 172.23.105.175. 2021-05-11 01:23:49,378 - root - INFO - Done with uninstall on 172.23.121.74. 2021-05-11 01:28:14,740 - root - INFO - Done with install on 172.23.107.45. 2021-05-11 01:28:15,380 - root - INFO - Done with install on 172.23.107.85. 2021-05-11 01:28:15,520 - root - INFO - Done with install on 172.23.107.107. 2021-05-11 01:28:15,612 - root - INFO - Done with install on 172.23.107.58. 2021-05-11 01:28:15,844 - root - INFO - Done with install on 172.23.107.84. 2021-05-11 01:28:15,947 - root - INFO - Done with install on 172.23.107.78. 2021-05-11 01:28:15,979 - root - INFO - Done with install on 172.23.107.47. 2021-05-11 01:28:16,107 - root - INFO - Done with install on 172.23.107.88. 2021-05-11 01:28:16,173 - root - INFO - Done with install on 172.23.107.91. 2021-05-11 01:28:16,236 - root - INFO - Done with install on 172.23.107.44. 2021-05-11 01:28:16,301 - root - INFO - Done with install on 172.23.107.54. 2021-05-11 01:28:16,382 - root - INFO - Done with install on 172.23.107.43. 2021-05-11 01:28:26,013 - root - INFO - Done with install on 172.23.106.250. 2021-05-11 01:28:26,657 - root - INFO - Done with install on 172.23.105.175. 2021-05-11 01:28:27,285 - root - INFO - Done with install on 172.23.106.251. 2021-05-11 01:28:29,489 - root - INFO - Done with install on 172.23.106.233. 2021-05-11 01:28:29,775 - root - INFO - Done with install on 172.23.121.78. 2021-05-11 01:28:29,964 - root - INFO - Done with install on 172.23.106.238. 2021-05-11 01:28:32,746 - root - INFO - Done with install on 172.23.106.236. 2021-05-11 01:28:35,973 - root - INFO - Done with install on 172.23.121.74. 2021-05-11 01:29:52,390 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.45 2021-05-11 01:29:52,391 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.45 2021-05-11 01:29:52,506 - root - INFO - Done with init on 172.23.107.45. 2021-05-11 01:29:52,526 - root - INFO - Done with cleanup on 172.23.107.45. 2021-05-11 01:29:52,668 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.85 2021-05-11 01:29:52,668 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.85 2021-05-11 01:29:52,792 - root - INFO - Done with init on 172.23.107.85. 2021-05-11 01:29:52,811 - root - INFO - Done with cleanup on 172.23.107.85. 2021-05-11 01:29:52,812 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.107 2021-05-11 01:29:52,812 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.107 2021-05-11 01:29:52,934 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.58 2021-05-11 01:29:52,936 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.58 2021-05-11 01:29:52,951 - root - INFO - Done with init on 172.23.107.107. 2021-05-11 01:29:52,971 - root - INFO - Done with cleanup on 172.23.107.107. 2021-05-11 01:29:53,115 - root - INFO - Done with init on 172.23.107.58. 2021-05-11 01:29:53,137 - root - INFO - Done with cleanup on 172.23.107.58. 2021-05-11 01:29:53,194 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.78 2021-05-11 01:29:53,196 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.78 2021-05-11 01:29:53,209 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.84 2021-05-11 01:29:53,209 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.84 2021-05-11 01:29:53,272 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.47 2021-05-11 01:29:53,273 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.47 2021-05-11 01:29:53,325 - root - INFO - Done with init on 172.23.107.78. 2021-05-11 01:29:53,328 - root - INFO - Done with init on 172.23.107.84. 2021-05-11 01:29:53,345 - root - INFO - Done with cleanup on 172.23.107.78. 2021-05-11 01:29:53,351 - root - INFO - Done with cleanup on 172.23.107.84. 2021-05-11 01:29:53,384 - root - INFO - Done with init on 172.23.107.47. 2021-05-11 01:29:53,403 - root - INFO - Done with cleanup on 172.23.107.47. 2021-05-11 01:29:53,437 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.88 2021-05-11 01:29:53,437 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.88 2021-05-11 01:29:53,476 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.91 2021-05-11 01:29:53,476 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.91 2021-05-11 01:29:53,529 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.44 2021-05-11 01:29:53,530 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.44 2021-05-11 01:29:53,555 - root - INFO - Done with init on 172.23.107.88. 2021-05-11 01:29:53,578 - root - INFO - Done with cleanup on 172.23.107.88. 2021-05-11 01:29:53,589 - root - INFO - Done with init on 172.23.107.91. 2021-05-11 01:29:53,611 - root - INFO - Done with cleanup on 172.23.107.91. 2021-05-11 01:29:53,644 - root - INFO - Done with init on 172.23.107.44. 2021-05-11 01:29:53,667 - root - INFO - Done with cleanup on 172.23.107.44. 2021-05-11 01:29:53,673 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.54 2021-05-11 01:29:53,674 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.54 2021-05-11 01:29:53,684 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.107.43 2021-05-11 01:29:53,684 - root - INFO - Setting KV memory quota as 256 MB on 172.23.107.43 2021-05-11 01:29:53,789 - root - INFO - Done with init on 172.23.107.54. 2021-05-11 01:29:53,805 - root - INFO - Done with init on 172.23.107.43. 2021-05-11 01:29:53,809 - root - INFO - Done with cleanup on 172.23.107.54. 2021-05-11 01:29:53,825 - root - INFO - Done with cleanup on 172.23.107.43. 2021-05-11 01:30:03,799 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.250 2021-05-11 01:30:03,801 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.250 01:30:03.927 [Jython-Netty-Client-8] ERROR org.python.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dorg.python.netty.leakDetectionLevel=advanced' or call ResourceLeakDetector.setLevel() 2021-05-11 01:30:03,931 - root - INFO - Done with init on 172.23.106.250. 2021-05-11 01:30:03,956 - root - INFO - Done with cleanup on 172.23.106.250. 2021-05-11 01:30:04,424 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.105.175 2021-05-11 01:30:04,424 - root - INFO - Setting KV memory quota as 256 MB on 172.23.105.175 2021-05-11 01:30:04,540 - root - INFO - Done with init on 172.23.105.175. 2021-05-11 01:30:04,563 - root - INFO - Done with cleanup on 172.23.105.175. 2021-05-11 01:30:05,115 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.251 2021-05-11 01:30:05,117 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.251 2021-05-11 01:30:05,243 - root - INFO - Done with init on 172.23.106.251. 2021-05-11 01:30:05,272 - root - INFO - Done with cleanup on 172.23.106.251. 2021-05-11 01:30:05,808 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.236 2021-05-11 01:30:05,808 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.236 2021-05-11 01:30:05,927 - root - INFO - Done with init on 172.23.106.236. 2021-05-11 01:30:05,950 - root - INFO - Done with cleanup on 172.23.106.236. 2021-05-11 01:30:07,391 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.121.78 2021-05-11 01:30:07,391 - root - INFO - Setting KV memory quota as 256 MB on 172.23.121.78 2021-05-11 01:30:07,394 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.233 2021-05-11 01:30:07,395 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.233 2021-05-11 01:30:07,515 - root - INFO - Done with init on 172.23.121.78. 2021-05-11 01:30:07,525 - root - INFO - Done with init on 172.23.106.233. 2021-05-11 01:30:07,539 - root - INFO - Done with cleanup on 172.23.121.78. 2021-05-11 01:30:07,548 - root - INFO - Done with cleanup on 172.23.106.233. 2021-05-11 01:30:07,851 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.238 2021-05-11 01:30:07,851 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.238 2021-05-11 01:30:07,970 - root - INFO - Done with init on 172.23.106.238. 2021-05-11 01:30:07,993 - root - INFO - Done with cleanup on 172.23.106.238. 2021-05-11 01:30:13,566 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.121.74 2021-05-11 01:30:13,568 - root - INFO - Setting KV memory quota as 256 MB on 172.23.121.74 2021-05-11 01:30:13,782 - root - INFO - Done with init on 172.23.121.74. 2021-05-11 01:30:13,806 - root - INFO - Done with cleanup on 172.23.121.74. 2021-05-11 01:30:31,177 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 01:30:31,194 - root - INFO - node:172.23.105.175:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,210 - root - INFO - node:172.23.106.233:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,221 - root - INFO - node:172.23.106.236:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,236 - root - INFO - node:172.23.106.238:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,250 - root - INFO - node:172.23.106.250:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,266 - root - INFO - node:172.23.106.251:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,280 - root - INFO - node:172.23.121.74:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,298 - root - INFO - node:172.23.121.78:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,312 - root - INFO - node:172.23.107.43:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,326 - root - INFO - node:172.23.107.58:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,345 - root - INFO - node:172.23.107.44:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,361 - root - INFO - node:172.23.107.45:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,380 - root - INFO - node:172.23.107.54:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,400 - root - INFO - node:172.23.107.47:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,424 - root - INFO - node:172.23.107.78:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,444 - root - INFO - node:172.23.107.84:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,463 - root - INFO - node:172.23.107.85:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,483 - root - INFO - node:172.23.107.88:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,497 - root - INFO - node:172.23.107.91:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,519 - root - INFO - node:172.23.107.107:8091 version:7.0.0-5127-enterprise aFamily:inet services:[u'kv'] 2021-05-11 01:30:31,519 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 01:30:31,520 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 01:30:31,520 - root - INFO - INSTALL COMPLETED ON: 172.23.105.175 2021-05-11 01:30:31,520 - root - INFO - INSTALL COMPLETED ON: 172.23.106.233 2021-05-11 01:30:31,522 - root - INFO - INSTALL COMPLETED ON: 172.23.106.236 2021-05-11 01:30:31,522 - root - INFO - INSTALL COMPLETED ON: 172.23.106.238 2021-05-11 01:30:31,523 - root - INFO - INSTALL COMPLETED ON: 172.23.106.250 2021-05-11 01:30:31,523 - root - INFO - INSTALL COMPLETED ON: 172.23.106.251 2021-05-11 01:30:31,523 - root - INFO - INSTALL COMPLETED ON: 172.23.121.74 2021-05-11 01:30:31,523 - root - INFO - INSTALL COMPLETED ON: 172.23.121.78 2021-05-11 01:30:31,523 - root - INFO - INSTALL COMPLETED ON: 172.23.107.43 2021-05-11 01:30:31,525 - root - INFO - INSTALL COMPLETED ON: 172.23.107.58 2021-05-11 01:30:31,525 - root - INFO - INSTALL COMPLETED ON: 172.23.107.44 2021-05-11 01:30:31,525 - root - INFO - INSTALL COMPLETED ON: 172.23.107.45 2021-05-11 01:30:31,525 - root - INFO - INSTALL COMPLETED ON: 172.23.107.54 2021-05-11 01:30:31,526 - root - INFO - INSTALL COMPLETED ON: 172.23.107.47 2021-05-11 01:30:31,526 - root - INFO - INSTALL COMPLETED ON: 172.23.107.78 2021-05-11 01:30:31,526 - root - INFO - INSTALL COMPLETED ON: 172.23.107.84 2021-05-11 01:30:31,526 - root - INFO - INSTALL COMPLETED ON: 172.23.107.85 2021-05-11 01:30:31,526 - root - INFO - INSTALL COMPLETED ON: 172.23.107.88 2021-05-11 01:30:31,526 - root - INFO - INSTALL COMPLETED ON: 172.23.107.91 2021-05-11 01:30:31,526 - root - INFO - INSTALL COMPLETED ON: 172.23.107.107 2021-05-11 01:30:31,528 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-11 01:30:31,533 - root - INFO - TOTAL INSTALL TIME = 470.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 8m 5s 1 actionable task: 1 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/20_node.ini -t volumetests.Metadata.volume.test_volume_taf,nodes_init=17,services_init=kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index,bucket_spec=single_bucket.metadata_volume,number_of_indexes_per_coll=10,quota_percent=80,kv_mem_quota=8000,index_mem_quota=22000 -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/20_node.ini -t volumetests.Metadata.volume.test_volume_taf,nodes_init=17,services_init=kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index,bucket_spec=single_bucket.metadata_volume,number_of_indexes_per_coll=10,quota_percent=80,kv_mem_quota=8000,index_mem_quota=22000 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/20_node.ini -t volumetests.Metadata.volume.test_volume_taf,nodes_init=17,services_init=kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index,bucket_spec=single_bucket.metadata_volume,number_of_indexes_per_coll=10,quota_percent=80,kv_mem_quota=8000,index_mem_quota=22000 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/20_node.ini -t volumetests.Metadata.volume.test_volume_taf,nodes_init=17,services_init=kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index,bucket_spec=single_bucket.metadata_volume,number_of_indexes_per_coll=10,quota_percent=80,kv_mem_quota=8000,index_mem_quota=22000 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/20_node.ini -t volumetests.Metadata.volume.test_volume_taf,nodes_init=17,services_init=kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index,bucket_spec=single_bucket.metadata_volume,number_of_indexes_per_coll=10,quota_percent=80,kv_mem_quota=8000,index_mem_quota=22000 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/20_node.ini -t volumetests.Metadata.volume.test_volume_taf,nodes_init=17,services_init=kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index,bucket_spec=single_bucket.metadata_volume,number_of_indexes_per_coll=10,quota_percent=80,kv_mem_quota=8000,index_mem_quota=22000 -m rest > Task :compileJava Note: /data/workspace/temp_durability_volume/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Global Test input params: {'cluster_name': '20_node', 'ini': '/tmp/20_node.ini', 'num_nodes': 20} Logs will be stored at /data/workspace/temp_durability_volume/logs/testrunner-21-May-11_01-30-42/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/20_node.ini -t volumetests.Metadata.volume.test_volume_taf,nodes_init=17,services_init=kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index,bucket_spec=single_bucket.metadata_volume,number_of_indexes_per_coll=10,quota_percent=80,kv_mem_quota=8000,index_mem_quota=22000' Test Input params: {'cluster_name': '20_node', 'kv_mem_quota': '8000', 'quota_percent': '80', 'ini': '/tmp/20_node.ini', 'number_of_indexes_per_coll': '10', 'num_nodes': 20, 'logs_folder': '/data/workspace/temp_durability_volume/logs/testrunner-21-May-11_01-30-42/test_1', 'rerun': False, 'index_mem_quota': '22000', 'nodes_init': '17', 'services_init': 'kv-kv-n1ql-index-index-index-index-index-index-index-index-index-index-index-index-index-index', 'bucket_spec': 'single_bucket.metadata_volume', 'case_number': 1} test_volume_taf (volumetests.Metadata.volume) ... 2021-05-11 01:30:57,237 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= volume setup started for test #1 test_volume_taf ========= 2021-05-11 01:30:57,999 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #1 test_volume_taf ========= 2021-05-11 01:31:00,102 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0x6 type=client open_count=1 channel=[id: 0x19abad65, 0.0.0.0/0.0.0.0:56171] timeout=300.0>) 2021-05-11 01:31:03,622 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0x7 type=client open_count=1 channel=[id: 0x6783c76b, 0.0.0.0/0.0.0.0:41201] timeout=300.0>) 2021-05-11 01:31:06,164 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0x8 type=client open_count=1 channel=[id: 0xa9e39baf, 0.0.0.0/0.0.0.0:34165] timeout=300.0>) 2021-05-11 01:31:08,194 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0x9 type=client open_count=1 channel=[id: 0x2960dc4f, 0.0.0.0/0.0.0.0:52926] timeout=300.0>) 2021-05-11 01:31:10,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0xa type=client open_count=1 channel=[id: 0x2a8e4904, 0.0.0.0/0.0.0.0:43268] timeout=300.0>) 2021-05-11 01:31:12,244 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0xb type=client open_count=1 channel=[id: 0xe0369950, 0.0.0.0/0.0.0.0:45747] timeout=300.0>) 2021-05-11 01:31:14,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0x1574f6cf, 0.0.0.0/0.0.0.0:51960] timeout=300.0>) 2021-05-11 01:31:19,767 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0x915ddb2e, 0.0.0.0/0.0.0.0:45052] timeout=300.0>) 2021-05-11 01:31:21,790 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x5cd4d73b, 0.0.0.0/0.0.0.0:44222] timeout=300.0>) 2021-05-11 01:31:23,816 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x38c84a35, 0.0.0.0/0.0.0.0:54607] timeout=300.0>) 2021-05-11 01:31:25,845 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x938dad54, 0.0.0.0/0.0.0.0:57375] timeout=300.0>) 2021-05-11 01:31:27,868 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x28e15881, 0.0.0.0/0.0.0.0:39260] timeout=300.0>) 2021-05-11 01:31:29,888 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xfa61e17e, 0.0.0.0/0.0.0.0:41402] timeout=300.0>) 2021-05-11 01:31:35,740 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x48bd1744, 0.0.0.0/0.0.0.0:59630] timeout=300.0>) 2021-05-11 01:31:37,756 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x11091aa5, 0.0.0.0/0.0.0.0:51767] timeout=300.0>) 2021-05-11 01:31:39,775 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0xfb374cf2, 0.0.0.0/0.0.0.0:42148] timeout=300.0>) 2021-05-11 01:31:41,790 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x69aa283e, 0.0.0.0/0.0.0.0:46117] timeout=300.0>) 2021-05-11 01:31:43,809 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0xfacc7344, 0.0.0.0/0.0.0.0:36422] timeout=300.0>) 2021-05-11 01:31:45,834 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0xb4459fac, 0.0.0.0/0.0.0.0:45775] timeout=300.0>) 2021-05-11 01:31:51,365 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x164230ec, 0.0.0.0/0.0.0.0:45991] timeout=300.0>) 2021-05-11 01:31:53,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x2f0e8feb, 0.0.0.0/0.0.0.0:57206] timeout=300.0>) 2021-05-11 01:31:55,400 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x2388e0d7, 0.0.0.0/0.0.0.0:36287] timeout=300.0>) 2021-05-11 01:31:57,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x15c4b0cc, 0.0.0.0/0.0.0.0:43811] timeout=300.0>) 2021-05-11 01:31:59,430 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x5b78724d, 0.0.0.0/0.0.0.0:34989] timeout=300.0>) 2021-05-11 01:32:01,450 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0x2cb4d9e0, 0.0.0.0/0.0.0.0:47667] timeout=300.0>) 2021-05-11 01:32:06,959 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x8b63dc0a, 0.0.0.0/0.0.0.0:55096] timeout=300.0>) 2021-05-11 01:32:08,973 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x8a04ad76, 0.0.0.0/0.0.0.0:53022] timeout=300.0>) 2021-05-11 01:32:10,992 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0xbd6201bf, 0.0.0.0/0.0.0.0:41947] timeout=300.0>) 2021-05-11 01:32:13,010 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x6284fc35, 0.0.0.0/0.0.0.0:42184] timeout=300.0>) 2021-05-11 01:32:15,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0x62651d0a, 0.0.0.0/0.0.0.0:36497] timeout=300.0>) 2021-05-11 01:32:17,043 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x610b00f6, 0.0.0.0/0.0.0.0:43695] timeout=300.0>) 2021-05-11 01:32:22,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0xcc45bf1e, 0.0.0.0/0.0.0.0:52725] timeout=300.0>) 2021-05-11 01:32:24,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0x31e214bf, 0.0.0.0/0.0.0.0:42510] timeout=300.0>) 2021-05-11 01:32:26,584 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0x464077db, 0.0.0.0/0.0.0.0:43733] timeout=300.0>) 2021-05-11 01:32:28,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0x603f54ff, 0.0.0.0/0.0.0.0:39779] timeout=300.0>) 2021-05-11 01:32:30,615 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0x93b21e14, 0.0.0.0/0.0.0.0:39895] timeout=300.0>) 2021-05-11 01:32:32,632 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x2a type=client open_count=1 channel=[id: 0xbbcb5edc, 0.0.0.0/0.0.0.0:46551] timeout=300.0>) 2021-05-11 01:32:38,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x2b type=client open_count=1 channel=[id: 0xe903db23, 0.0.0.0/0.0.0.0:55812] timeout=300.0>) 2021-05-11 01:32:40,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x2c type=client open_count=1 channel=[id: 0x9164c240, 0.0.0.0/0.0.0.0:41402] timeout=300.0>) 2021-05-11 01:32:42,426 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x2d type=client open_count=1 channel=[id: 0x45f1d569, 0.0.0.0/0.0.0.0:48787] timeout=300.0>) 2021-05-11 01:32:44,444 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x2e type=client open_count=1 channel=[id: 0x6adfef00, 0.0.0.0/0.0.0.0:58893] timeout=300.0>) 2021-05-11 01:32:46,460 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x2f type=client open_count=1 channel=[id: 0xcb4056ac, 0.0.0.0/0.0.0.0:33704] timeout=300.0>) 2021-05-11 01:32:48,477 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x30 type=client open_count=1 channel=[id: 0x250c7689, 0.0.0.0/0.0.0.0:36727] timeout=300.0>) 2021-05-11 01:32:53,957 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x31 type=client open_count=1 channel=[id: 0xd7b05881, 0.0.0.0/0.0.0.0:53667] timeout=300.0>) 2021-05-11 01:32:55,976 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x32 type=client open_count=1 channel=[id: 0xea80ae9e, 0.0.0.0/0.0.0.0:35969] timeout=300.0>) 2021-05-11 01:32:57,990 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x33 type=client open_count=1 channel=[id: 0xc5b37216, 0.0.0.0/0.0.0.0:59203] timeout=300.0>) 2021-05-11 01:33:00,009 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x34 type=client open_count=1 channel=[id: 0x106edfa7, 0.0.0.0/0.0.0.0:60926] timeout=300.0>) 2021-05-11 01:33:02,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x35 type=client open_count=1 channel=[id: 0xcf39c16a, 0.0.0.0/0.0.0.0:45475] timeout=300.0>) 2021-05-11 01:33:04,042 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.43:8091 during connect (<_realsocket at 0x36 type=client open_count=1 channel=[id: 0x3e5b1dd5, 0.0.0.0/0.0.0.0:46110] timeout=300.0>) 2021-05-11 01:33:09,273 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.43:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.43:8091 during connect (<_realsocket at 0x37 type=client open_count=1 channel=[id: 0x7b0fae95, 0.0.0.0/0.0.0.0:54745] timeout=300.0>) 2021-05-11 01:33:11,289 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.43:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.43:8091 during connect (<_realsocket at 0x38 type=client open_count=1 channel=[id: 0x66cb3f5b, 0.0.0.0/0.0.0.0:42229] timeout=300.0>) 2021-05-11 01:33:13,303 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.43:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.43:8091 during connect (<_realsocket at 0x39 type=client open_count=1 channel=[id: 0x3b7e3906, 0.0.0.0/0.0.0.0:57959] timeout=300.0>) 2021-05-11 01:33:15,321 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.43:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.43:8091 during connect (<_realsocket at 0x3a type=client open_count=1 channel=[id: 0x8427c457, 0.0.0.0/0.0.0.0:52783] timeout=300.0>) 2021-05-11 01:33:17,334 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.43:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.43:8091 during connect (<_realsocket at 0x3b type=client open_count=1 channel=[id: 0xe758109e, 0.0.0.0/0.0.0.0:53811] timeout=300.0>) 2021-05-11 01:33:19,349 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.43:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.58:8091 during connect (<_realsocket at 0x3c type=client open_count=1 channel=[id: 0x46575437, 0.0.0.0/0.0.0.0:41717] timeout=300.0>) 2021-05-11 01:33:24,684 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.58:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.58:8091 during connect (<_realsocket at 0x3d type=client open_count=1 channel=[id: 0xbe440d7f, 0.0.0.0/0.0.0.0:48278] timeout=300.0>) 2021-05-11 01:33:26,703 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.58:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.58:8091 during connect (<_realsocket at 0x3e type=client open_count=1 channel=[id: 0x45ffc166, 0.0.0.0/0.0.0.0:47209] timeout=300.0>) 2021-05-11 01:33:28,723 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.58:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.58:8091 during connect (<_realsocket at 0x3f type=client open_count=1 channel=[id: 0xf65af4ac, 0.0.0.0/0.0.0.0:42654] timeout=300.0>) 2021-05-11 01:33:30,739 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.58:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.58:8091 during connect (<_realsocket at 0x40 type=client open_count=1 channel=[id: 0x185780bc, 0.0.0.0/0.0.0.0:57816] timeout=300.0>) 2021-05-11 01:33:32,753 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.58:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.58:8091 during connect (<_realsocket at 0x41 type=client open_count=1 channel=[id: 0xacbb2f1c, 0.0.0.0/0.0.0.0:52288] timeout=300.0>) 2021-05-11 01:33:34,769 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.58:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.44:8091 during connect (<_realsocket at 0x42 type=client open_count=1 channel=[id: 0x36c8bf96, 0.0.0.0/0.0.0.0:49155] timeout=300.0>) 2021-05-11 01:33:40,068 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.44:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.44:8091 during connect (<_realsocket at 0x43 type=client open_count=1 channel=[id: 0x3568e232, 0.0.0.0/0.0.0.0:58358] timeout=300.0>) 2021-05-11 01:33:42,082 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.44:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.44:8091 during connect (<_realsocket at 0x44 type=client open_count=1 channel=[id: 0x80653266, 0.0.0.0/0.0.0.0:51847] timeout=300.0>) 2021-05-11 01:33:44,098 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.44:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.44:8091 during connect (<_realsocket at 0x45 type=client open_count=1 channel=[id: 0x329277fd, 0.0.0.0/0.0.0.0:50183] timeout=300.0>) 2021-05-11 01:33:46,117 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.44:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.44:8091 during connect (<_realsocket at 0x46 type=client open_count=1 channel=[id: 0x1a6ced5d, 0.0.0.0/0.0.0.0:54133] timeout=300.0>) 2021-05-11 01:33:48,131 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.44:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.44:8091 during connect (<_realsocket at 0x47 type=client open_count=1 channel=[id: 0x47668268, 0.0.0.0/0.0.0.0:56133] timeout=300.0>) 2021-05-11 01:33:50,147 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.44:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.45:8091 during connect (<_realsocket at 0x48 type=client open_count=1 channel=[id: 0xe1f4fcb3, 0.0.0.0/0.0.0.0:58527] timeout=300.0>) 2021-05-11 01:33:55,450 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.45:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.45:8091 during connect (<_realsocket at 0x49 type=client open_count=1 channel=[id: 0x4cd1cafc, 0.0.0.0/0.0.0.0:42248] timeout=300.0>) 2021-05-11 01:33:57,464 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.45:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.45:8091 during connect (<_realsocket at 0x4a type=client open_count=1 channel=[id: 0x8e01e599, 0.0.0.0/0.0.0.0:59261] timeout=300.0>) 2021-05-11 01:33:59,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.45:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.45:8091 during connect (<_realsocket at 0x4b type=client open_count=1 channel=[id: 0xe14126a1, 0.0.0.0/0.0.0.0:60891] timeout=300.0>) 2021-05-11 01:34:01,489 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.45:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.45:8091 during connect (<_realsocket at 0x4c type=client open_count=1 channel=[id: 0x1be1dade, 0.0.0.0/0.0.0.0:50742] timeout=300.0>) 2021-05-11 01:34:03,506 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.45:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.45:8091 during connect (<_realsocket at 0x4d type=client open_count=1 channel=[id: 0xfa27d379, 0.0.0.0/0.0.0.0:52080] timeout=300.0>) 2021-05-11 01:34:05,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.45:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.54:8091 during connect (<_realsocket at 0x4e type=client open_count=1 channel=[id: 0xd30322d4, 0.0.0.0/0.0.0.0:52903] timeout=300.0>) 2021-05-11 01:34:10,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.54:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.54:8091 during connect (<_realsocket at 0x4f type=client open_count=1 channel=[id: 0x8c146f03, 0.0.0.0/0.0.0.0:49515] timeout=300.0>) 2021-05-11 01:34:12,897 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.54:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.54:8091 during connect (<_realsocket at 0x50 type=client open_count=1 channel=[id: 0xc743a793, 0.0.0.0/0.0.0.0:58528] timeout=300.0>) 2021-05-11 01:34:14,911 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.54:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.54:8091 during connect (<_realsocket at 0x51 type=client open_count=1 channel=[id: 0x1306fe31, 0.0.0.0/0.0.0.0:51880] timeout=300.0>) 2021-05-11 01:34:16,928 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.54:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.54:8091 during connect (<_realsocket at 0x52 type=client open_count=1 channel=[id: 0x533f08a9, 0.0.0.0/0.0.0.0:39218] timeout=300.0>) 2021-05-11 01:34:18,940 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.54:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.54:8091 during connect (<_realsocket at 0x53 type=client open_count=1 channel=[id: 0xba6e2791, 0.0.0.0/0.0.0.0:55477] timeout=300.0>) 2021-05-11 01:34:20,954 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.54:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.47:8091 during connect (<_realsocket at 0x54 type=client open_count=1 channel=[id: 0xa6120ab9, 0.0.0.0/0.0.0.0:45903] timeout=300.0>) 2021-05-11 01:34:26,279 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.47:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.47:8091 during connect (<_realsocket at 0x55 type=client open_count=1 channel=[id: 0xd964f564, 0.0.0.0/0.0.0.0:37260] timeout=300.0>) 2021-05-11 01:34:28,299 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.47:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.47:8091 during connect (<_realsocket at 0x56 type=client open_count=1 channel=[id: 0xd527f84b, 0.0.0.0/0.0.0.0:40885] timeout=300.0>) 2021-05-11 01:34:30,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.47:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.47:8091 during connect (<_realsocket at 0x57 type=client open_count=1 channel=[id: 0x117ff0a2, 0.0.0.0/0.0.0.0:50785] timeout=300.0>) 2021-05-11 01:34:32,332 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.47:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.47:8091 during connect (<_realsocket at 0x58 type=client open_count=1 channel=[id: 0x09d5a5fe, 0.0.0.0/0.0.0.0:35739] timeout=300.0>) 2021-05-11 01:34:34,346 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.47:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.47:8091 during connect (<_realsocket at 0x59 type=client open_count=1 channel=[id: 0x2e5e24db, 0.0.0.0/0.0.0.0:59891] timeout=300.0>) 2021-05-11 01:34:36,362 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.47:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.78:8091 during connect (<_realsocket at 0x5a type=client open_count=1 channel=[id: 0x7fa1a700, 0.0.0.0/0.0.0.0:57436] timeout=300.0>) 2021-05-11 01:34:41,673 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.78:8091 during connect (<_realsocket at 0x5b type=client open_count=1 channel=[id: 0x761360f9, 0.0.0.0/0.0.0.0:53871] timeout=300.0>) 2021-05-11 01:34:43,690 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.78:8091 during connect (<_realsocket at 0x5c type=client open_count=1 channel=[id: 0x49305531, 0.0.0.0/0.0.0.0:37561] timeout=300.0>) 2021-05-11 01:34:45,703 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.78:8091 during connect (<_realsocket at 0x5d type=client open_count=1 channel=[id: 0x1b6f2c6d, 0.0.0.0/0.0.0.0:52868] timeout=300.0>) 2021-05-11 01:34:47,720 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.78:8091 during connect (<_realsocket at 0x5e type=client open_count=1 channel=[id: 0x58853a95, 0.0.0.0/0.0.0.0:60186] timeout=300.0>) 2021-05-11 01:34:49,739 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.78:8091 during connect (<_realsocket at 0x5f type=client open_count=1 channel=[id: 0x0068eecd, 0.0.0.0/0.0.0.0:34261] timeout=300.0>) 2021-05-11 01:34:51,757 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.84:8091 during connect (<_realsocket at 0x60 type=client open_count=1 channel=[id: 0x4d7a1881, 0.0.0.0/0.0.0.0:53810] timeout=300.0>) 2021-05-11 01:34:57,069 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.84:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.84:8091 during connect (<_realsocket at 0x61 type=client open_count=1 channel=[id: 0x77526bf4, 0.0.0.0/0.0.0.0:35041] timeout=300.0>) 2021-05-11 01:34:59,082 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.84:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.84:8091 during connect (<_realsocket at 0x62 type=client open_count=1 channel=[id: 0x93da6d4a, 0.0.0.0/0.0.0.0:42511] timeout=300.0>) 2021-05-11 01:35:01,098 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.84:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.84:8091 during connect (<_realsocket at 0x63 type=client open_count=1 channel=[id: 0x80896914, 0.0.0.0/0.0.0.0:56083] timeout=300.0>) 2021-05-11 01:35:03,115 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.84:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.84:8091 during connect (<_realsocket at 0x64 type=client open_count=1 channel=[id: 0xce467853, 0.0.0.0/0.0.0.0:37820] timeout=300.0>) 2021-05-11 01:35:05,128 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.84:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.84:8091 during connect (<_realsocket at 0x65 type=client open_count=1 channel=[id: 0x2021447b, 0.0.0.0/0.0.0.0:51146] timeout=300.0>) 2021-05-11 01:35:07,144 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.84:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.85:8091 during connect (<_realsocket at 0x66 type=client open_count=1 channel=[id: 0xec8fb351, 0.0.0.0/0.0.0.0:46491] timeout=300.0>) 2021-05-11 01:35:12,437 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.85:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.85:8091 during connect (<_realsocket at 0x67 type=client open_count=1 channel=[id: 0x1f83b2d1, 0.0.0.0/0.0.0.0:35468] timeout=300.0>) 2021-05-11 01:35:14,451 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.85:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.85:8091 during connect (<_realsocket at 0x68 type=client open_count=1 channel=[id: 0xf93f077a, 0.0.0.0/0.0.0.0:32865] timeout=300.0>) 2021-05-11 01:35:16,466 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.85:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.85:8091 during connect (<_realsocket at 0x69 type=client open_count=1 channel=[id: 0x67748711, 0.0.0.0/0.0.0.0:51331] timeout=300.0>) 2021-05-11 01:35:18,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.85:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.85:8091 during connect (<_realsocket at 0x6a type=client open_count=1 channel=[id: 0x743709ca, 0.0.0.0/0.0.0.0:56749] timeout=300.0>) 2021-05-11 01:35:20,489 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.85:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.85:8091 during connect (<_realsocket at 0x6b type=client open_count=1 channel=[id: 0x86382768, 0.0.0.0/0.0.0.0:47612] timeout=300.0>) 2021-05-11 01:35:22,506 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.85:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.88:8091 during connect (<_realsocket at 0x6c type=client open_count=1 channel=[id: 0x190bf165, 0.0.0.0/0.0.0.0:50789] timeout=300.0>) 2021-05-11 01:35:27,857 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.88:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.88:8091 during connect (<_realsocket at 0x6d type=client open_count=1 channel=[id: 0xe251a048, 0.0.0.0/0.0.0.0:54475] timeout=300.0>) 2021-05-11 01:35:29,868 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.88:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.88:8091 during connect (<_realsocket at 0x6e type=client open_count=1 channel=[id: 0x54ff6f8d, 0.0.0.0/0.0.0.0:49575] timeout=300.0>) 2021-05-11 01:35:31,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.88:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.88:8091 during connect (<_realsocket at 0x6f type=client open_count=1 channel=[id: 0xa29f585c, 0.0.0.0/0.0.0.0:41713] timeout=300.0>) 2021-05-11 01:35:33,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.88:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.88:8091 during connect (<_realsocket at 0x70 type=client open_count=1 channel=[id: 0x2744017a, 0.0.0.0/0.0.0.0:56013] timeout=300.0>) 2021-05-11 01:35:35,904 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.88:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.88:8091 during connect (<_realsocket at 0x71 type=client open_count=1 channel=[id: 0x4530fb74, 0.0.0.0/0.0.0.0:57686] timeout=300.0>) 2021-05-11 01:35:37,915 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.88:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.91:8091 during connect (<_realsocket at 0x72 type=client open_count=1 channel=[id: 0x98dde610, 0.0.0.0/0.0.0.0:56759] timeout=300.0>) 2021-05-11 01:35:43,076 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.91:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.91:8091 during connect (<_realsocket at 0x73 type=client open_count=1 channel=[id: 0x9ac5ac47, 0.0.0.0/0.0.0.0:42805] timeout=300.0>) 2021-05-11 01:35:45,089 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.91:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.91:8091 during connect (<_realsocket at 0x74 type=client open_count=1 channel=[id: 0x0e4ef89d, 0.0.0.0/0.0.0.0:37429] timeout=300.0>) 2021-05-11 01:35:47,107 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.91:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.91:8091 during connect (<_realsocket at 0x75 type=client open_count=1 channel=[id: 0x34d44afd, 0.0.0.0/0.0.0.0:48639] timeout=300.0>) 2021-05-11 01:35:49,121 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.91:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.91:8091 during connect (<_realsocket at 0x76 type=client open_count=1 channel=[id: 0x0cafac0a, 0.0.0.0/0.0.0.0:53022] timeout=300.0>) 2021-05-11 01:35:51,134 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.91:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.91:8091 during connect (<_realsocket at 0x77 type=client open_count=1 channel=[id: 0x614ef8f3, 0.0.0.0/0.0.0.0:54296] timeout=300.0>) 2021-05-11 01:35:53,150 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.91:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.107:8091 during connect (<_realsocket at 0x78 type=client open_count=1 channel=[id: 0x4a3639cf, 0.0.0.0/0.0.0.0:60606] timeout=300.0>) 2021-05-11 01:35:58,502 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.107:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.107:8091 during connect (<_realsocket at 0x79 type=client open_count=1 channel=[id: 0x82acae24, 0.0.0.0/0.0.0.0:49679] timeout=300.0>) 2021-05-11 01:36:00,515 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.107:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.107:8091 during connect (<_realsocket at 0x7a type=client open_count=1 channel=[id: 0xab229be0, 0.0.0.0/0.0.0.0:32818] timeout=300.0>) 2021-05-11 01:36:02,528 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.107:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.107:8091 during connect (<_realsocket at 0x7b type=client open_count=1 channel=[id: 0x71e9816f, 0.0.0.0/0.0.0.0:57513] timeout=300.0>) 2021-05-11 01:36:04,548 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.107:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.107:8091 during connect (<_realsocket at 0x7c type=client open_count=1 channel=[id: 0xb45343ee, 0.0.0.0/0.0.0.0:36155] timeout=300.0>) 2021-05-11 01:36:06,562 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.107:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.107:8091 during connect (<_realsocket at 0x7d type=client open_count=1 channel=[id: 0x3fc08017, 0.0.0.0/0.0.0.0:60631] timeout=300.0>) 2021-05-11 01:36:08,576 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.107:8091/nodes/self error [Errno 111] Connection refused 2021-05-11 01:38:51,561 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-11 01:39:11,693 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.105.175, 172.23.106.233, 172.23.106.236, 172.23.106.238, 172.23.106.250, 172.23.106.251, 172.23.121.74, 172.23.121.78, 172.23.107.43, 172.23.107.58, 172.23.107.44, 172.23.107.45, 172.23.107.54, 172.23.107.47, 172.23.107.78, 172.23.107.84, 172.23.107.85, 172.23.107.88, 172.23.107.91, 172.23.107.107 initialized 2021-05-11 01:39:11,710 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #1 test_volume_taf ========= 2021-05-11 01:39:11,730 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #1 test_volume_taf ========= 2021-05-11 01:41:19,924 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+-----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-----------+-----------------------+----------------+--------------+ | 172.23.105.175 | kv | 7.0.0-5127-enterprise | 0.162948107295 | Cluster node | | 172.23.106.233 | ['kv'] | | | <--- IN --- | | 172.23.106.236 | ['n1ql'] | | | <--- IN --- | | 172.23.106.238 | ['index'] | | | <--- IN --- | | 172.23.106.250 | ['index'] | | | <--- IN --- | | 172.23.106.251 | ['index'] | | | <--- IN --- | | 172.23.121.74 | ['index'] | | | <--- IN --- | | 172.23.121.78 | ['index'] | | | <--- IN --- | | 172.23.107.43 | ['index'] | | | <--- IN --- | | 172.23.107.58 | ['index'] | | | <--- IN --- | | 172.23.107.44 | ['index'] | | | <--- IN --- | | 172.23.107.45 | ['index'] | | | <--- IN --- | | 172.23.107.54 | ['index'] | | | <--- IN --- | | 172.23.107.47 | ['index'] | | | <--- IN --- | | 172.23.107.78 | ['index'] | | | <--- IN --- | | 172.23.107.84 | ['index'] | | | <--- IN --- | | 172.23.107.85 | ['index'] | | | <--- IN --- | +----------------+-----------+-----------------------+----------------+--------------+ 2021-05-11 01:41:19,947 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-11 01:41:30,006 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-11 01:41:30,039 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 10.117000103 sec 2021-05-11 01:41:30,226 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-11 01:42:14,266 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-11 01:42:14,362 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------------+-----------+----------+------------+-----+-------+------------+-----------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------------+-----------+----------+------------+-----+-------+------------+-----------+------------+ | s9NLzpdOOFpz3X%--30-262000 | couchbase | 1 | none | 0 | 0 | 4194304000 | 343841880 | 1636987803 | +----------------------------+-----------+----------+------------+-----+-------+------------+-----------+------------+ 2021-05-11 01:42:14,362 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-11 01:42:25,073 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-11 01:42:25,092 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-11 01:42:29,367 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-------------+-------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-------------+-------------+-------------------------+-------------------+-----------------------+ | 172.23.107.58 | index | 2.04799597939 | 25084014592 | 23995346944 | 228618240 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.44 | index | 2.28413654618 | 25084014592 | 24029282304 | 137818112 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.105.175 | kv | 23.0442614514 | 25084059648 | 23721021440 | 0 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.45 | index | 2.3197492163 | 25084010496 | 24038137856 | 118796288 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.47 | index | 2.34541577825 | 25084014592 | 24025083904 | 121827328 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.84 | index | 1.85906293179 | 25084014592 | 24013340672 | 126259200 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.121.78 | index | 2.22501571339 | 25084071936 | 23956066304 | 1089036288 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.106.250 | index | 2.18675380168 | 25084014592 | 23996690432 | 80904192 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.43 | index | 2.37050043898 | 25084014592 | 24000823296 | 964407296 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.106.236 | n1ql | 1.37982940291 | 25084067840 | 23989178368 | 0 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.85 | index | 1.958814666 | 25084010496 | 24001433600 | 112365568 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.106.251 | index | 2.2758707406 | 25084014592 | 23987531776 | 71049216 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.106.233 | kv | 17.8525886254 | 25084071936 | 23769669632 | 0 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.106.238 | index | 2.57667169432 | 25084071936 | 23992737792 | 74969088 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.54 | index | 2.39498432602 | 25084014592 | 24012529664 | 418398208 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.107.78 | index | 2.11938801104 | 25084014592 | 24012263424 | 110862336 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | | 172.23.121.74 | index | 2.11241041117 | 25084071936 | 23970152448 | 214093824 / 3758092288 | 0 / 0 | 7.0.0-5127-enterprise | +----------------+----------+-----------------+-------------+-------------+-------------------------+-------------------+-----------------------+ 2021-05-11 01:42:31,739 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-11 01:43:42,882 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | s9NLzpdOOFpz3X%--30-262000 | couchbase | 1 | none | 0 | 10000 | 4194304000 | 266983632 | 434381289 | +----------------------------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-11 01:43:42,884 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #1 test_volume_taf ========= 2021-05-11 01:43:58,200 | test | INFO | MainThread | [Metadata:create_indexes:133] Creating indexes with defer build 2021-05-11 04:56:16,259 | test | ERROR | MainThread | [rest_client:_http_request:748] POST http://172.23.106.236:8093/query?statement=CREATE+INDEX+%60gsi6221%60+ON+%60s9NLzpdOOFpz3X%25--30-262000%60.%60y--30-347000%60.%60JaMXMS2HcvZIp%25hXEo-30-354000%60%28%60age%60%29+WITH+%7B+%27defer_build%27%3A+true%2C+%27num_replica%27%3A+0+%7D body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown { "requestID": "9e9cf0bf-cd7d-4338-ab65-4c15823440ce", "signature": null, "results": [ ], "errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Index creation for index gsi6221, bucket s9NLzpdOOFpz3X%--30-262000, scope y--30-347000, collection JaMXMS2HcvZIp%hXEo-30-354000 cannot start. Reason: Create index or Alter replica cannot proceed due to network partition, node failover or indexer failure.."}], "status": "errors", "metrics": {"elapsedTime": "3.641195108s","executionTime": "3.641139289s","resultCount": 0,"resultSize": 0,"serviceLoad": 3,"errorCount": 1} } auth: Administrator:password 2021-05-11 05:04:16,536 | test | ERROR | MainThread | [rest_client:_http_request:748] POST http://172.23.106.236:8093/query?statement=CREATE+INDEX+%60gsi6334%60+ON+%60s9NLzpdOOFpz3X%25--30-262000%60.%60y--30-347000%60.%60OUy99tVZX9lP_-30-349000%60%28%60age%60%29+WITH+%7B+%27defer_build%27%3A+true%2C+%27num_replica%27%3A+0+%7D body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown { "requestID": "9b505c3e-86c4-48bd-b677-eaef45480f49", "signature": null, "results": [ ], "errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Index creation for index gsi6334, bucket s9NLzpdOOFpz3X%--30-262000, scope y--30-347000, collection OUy99tVZX9lP_-30-349000 cannot start. Reason: Create index or Alter replica cannot proceed due to network partition, node failover or indexer failure.."}], "status": "errors", "metrics": {"elapsedTime": "3.276888589s","executionTime": "3.276829092s","resultCount": 0,"resultSize": 0,"serviceLoad": 3,"errorCount": 1} } auth: Administrator:password 2021-05-11 05:04:46,644 | test | ERROR | MainThread | [rest_client:_http_request:748] POST http://172.23.106.236:8093/query?statement=CREATE+INDEX+%60gsi6338%60+ON+%60s9NLzpdOOFpz3X%25--30-262000%60.%60y--30-347000%60.%60OUy99tVZX9lP_-30-349000%60%28%60age%60%29+WITH+%7B+%27defer_build%27%3A+true%2C+%27num_replica%27%3A+0+%7D body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown { "requestID": "edbf1e69-e1b5-4cde-9d08-d7999c3bea3c", "signature": null, "results": [ ], "errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Index creation for index gsi6338, bucket s9NLzpdOOFpz3X%--30-262000, scope y--30-347000, collection OUy99tVZX9lP_-30-349000 cannot start. Reason: Create index or Alter replica cannot proceed due to network partition, node failover or indexer failure.."}], "status": "errors", "metrics": {"elapsedTime": "4.666457651s","executionTime": "4.666394045s","resultCount": 0,"resultSize": 0,"serviceLoad": 3,"errorCount": 1} } auth: Administrator:password 2021-05-11 05:05:26,213 | test | ERROR | MainThread | [rest_client:_http_request:748] POST http://172.23.106.236:8093/query?statement=CREATE+INDEX+%60gsi6342%60+ON+%60s9NLzpdOOFpz3X%25--30-262000%60.%60y--30-347000%60.%60kh-30-352000%60%28%60age%60%29+WITH+%7B+%27defer_build%27%3A+true%2C+%27num_replica%27%3A+0+%7D body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown { "requestID": "5ad7eef0-c140-43fe-98cd-332961923563", "signature": null, "results": [ ], "errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Index creation for index gsi6342, bucket s9NLzpdOOFpz3X%--30-262000, scope y--30-347000, collection kh-30-352000 cannot start. Reason: Create index or Alter replica cannot proceed due to network partition, node failover or indexer failure.."}], "status": "errors", "metrics": {"elapsedTime": "2.822245273s","executionTime": "2.822178419s","resultCount": 0,"resultSize": 0,"serviceLoad": 3,"errorCount": 1} } auth: Administrator:password 2021-05-11 05:05:43,348 | test | ERROR | MainThread | [rest_client:_http_request:748] POST http://172.23.106.236:8093/query?statement=CREATE+INDEX+%60gsi6344%60+ON+%60s9NLzpdOOFpz3X%25--30-262000%60.%60y--30-347000%60.%60kh-30-352000%60%28%60age%60%29+WITH+%7B+%27defer_build%27%3A+true%2C+%27num_replica%27%3A+0+%7D body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown { "requestID": "f7adb6c5-8fd2-496e-9990-066e3892e0a1", "signature": null, "results": [ ], "errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Index creation for index gsi6344, bucket s9NLzpdOOFpz3X%--30-262000, scope y--30-347000, collection kh-30-352000 cannot start. Reason: Create index or Alter replica cannot proceed due to network partition, node failover or indexer failure.."}], "status": "errors", "metrics": {"elapsedTime": "1.407716775s","executionTime": "1.40765048s","resultCount": 0,"resultSize": 0,"serviceLoad": 3,"errorCount": 1} } auth: Administrator:password 2021-05-11 05:06:29,279 | test | ERROR | MainThread | [rest_client:_http_request:748] POST http://172.23.106.236:8093/query?statement=CREATE+INDEX+%60gsi6351%60+ON+%60s9NLzpdOOFpz3X%25--30-262000%60.%60y--30-347000%60.%60DY7l-WAP-30-355000%60%28%60age%60%29+WITH+%7B+%27defer_build%27%3A+true%2C+%27num_replica%27%3A+0+%7D body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown { "requestID": "b52ed60c-3f21-4a14-9229-6e59c325966d", "signature": null, "results": [ ], "errors": [{"code":5000,"msg":"GSI CreateIndex() - cause: Index creation for index gsi6351, bucket s9NLzpdOOFpz3X%--30-262000, scope y--30-347000, collection DY7l-WAP-30-355000 cannot start. Reason: Create index or Alter replica cannot proceed due to network partition, node failover or indexer failure.."}], "status": "errors", "metrics": {"elapsedTime": "4.160213096s","executionTime": "4.160144359s","resultCount": 0,"resultSize": 0,"serviceLoad": 3,"errorCount": 1} } auth: Administrator:password