Started by user Asad Zaidi [EnvInject] - Loading node environment variables. Building remotely on slv-sc2402-32g-12c (volume_test_collections) in workspace /data/workspace/temp_backup_service_volume [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_backup_service_volume # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision c844cdb7d7386fbf96a4aecf562319407b5c9e63 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f c844cdb7d7386fbf96a4aecf562319407b5c9e63 > /usr/bin/git rev-list c844cdb7d7386fbf96a4aecf562319407b5c9e63 # timeout=10 [temp_backup_service_volume] $ /bin/sh -xe /tmp/jenkins2981051074381208490.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 data_path:/data/data/ [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 [_1] ip:172.23.105.175 [_2] ip:172.23.106.233 [_3] ip:172.23.106.236 [_4] ip:172.23.106.238 [_5] ip:172.23.106.250 [_6] ip:172.23.106.251 [_7] ip:172.23.121.74 [_8] ip:172.23.121.78' + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 data_path:/data/data/ [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 9:_9 10:_10 11:_11 12:_12 13:_13 14:_14 15:_15 [_1] ip:172.23.105.175 [_2] ip:172.23.106.233 [_3] ip:172.23.106.236 [_4] ip:172.23.106.238 [_5] ip:172.23.106.250 [_6] ip:172.23.106.251 [_7] ip:172.23.121.74 [_8] ip:172.23.136.148 [_9] ip:172.23.136.149 [_10] ip:172.23.136.135 [_11] ip:172.23.136.136 [_12] ip:172.23.136.144 [_13] ip:172.23.136.145 [_14] ip:172.23.136.146 [_15] ip:172.23.136.147 ' + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 data_path:/data/ [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 9:_9 10:_10 11:_11 12:_12 [_1] ip:172.23.97.215 [_2] ip:172.23.97.232 [_3] ip:172.23.97.237 [_4] ip:172.23.97.235 [_5] ip:172.23.97.236 [_6] ip:172.23.97.227 [_7] ip:172.23.97.216 [_8] ip:172.23.97.217 [_9] ip:172.23.107.5 [_10] ip:172.23.107.2 [_11] ip:172.23.107.3 [_12] ip:172.23.107.4 ' + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + cp guides/gradle.properties . + git fetch http://review.couchbase.org/TAF refs/changes/71/150171/7 From http://review.couchbase.org/TAF * branch refs/changes/71/150171/7 -> FETCH_HEAD + git cherry-pick FETCH_HEAD [detached HEAD ba4a1b9] [Volume Testing] Configuration change Author: Asad Zaidi 2 files changed, 14 insertions(+), 5 deletions(-) [temp_backup_service_volume] $ /bin/sh -xe /tmp/jenkins2428140110847142964.sh + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + guides/gradlew --refresh-dependencies install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p version=7.0.0-4907,product=cb,parallel=True' Starting a Gradle Daemon, 52 busy and 1 stopped Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p version=7.0.0-4907,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p version=7.0.0-4907,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p version=7.0.0-4907,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p version=7.0.0-4907,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p version=7.0.0-4907,product=cb,parallel=True > Task :install ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2 2021-04-14 06:20:30,926 - root - INFO - Done with uninstall on 172.23.106.250. 2021-04-14 06:20:31,719 - root - INFO - Done with uninstall on 172.23.106.236. 2021-04-14 06:20:32,575 - root - INFO - Done with uninstall on 172.23.106.251. 2021-04-14 06:20:37,119 - root - INFO - Done with uninstall on 172.23.121.78. 2021-04-14 06:21:41,562 - root - INFO - Done with uninstall on 172.23.105.175. 2021-04-14 06:21:41,739 - root - INFO - Done with uninstall on 172.23.106.238. 2021-04-14 06:21:41,799 - root - INFO - Done with uninstall on 172.23.106.233. 2021-04-14 06:21:41,914 - root - INFO - Done with uninstall on 172.23.121.74. 2021-04-14 06:25:11,697 - root - INFO - Done with install on 172.23.106.250. 2021-04-14 06:25:12,092 - root - INFO - Done with install on 172.23.106.236. 2021-04-14 06:25:13,303 - root - INFO - Done with install on 172.23.106.251. 2021-04-14 06:25:17,618 - root - INFO - Done with install on 172.23.121.78. 2021-04-14 06:26:17,802 - root - INFO - Setting KV memory quota as 12821 MB on 172.23.106.251 2021-04-14 06:26:17,923 - root - INFO - Done with init on 172.23.106.251. 2021-04-14 06:26:17,947 - root - INFO - Done with cleanup on 172.23.106.251. 2021-04-14 06:26:21,213 - root - INFO - Done with install on 172.23.105.175. 2021-04-14 06:26:22,332 - root - INFO - Done with install on 172.23.106.238. 2021-04-14 06:26:22,809 - root - INFO - Done with install on 172.23.106.233. 2021-04-14 06:26:28,078 - root - INFO - Done with install on 172.23.121.74. 2021-04-14 06:26:44,538 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.250 2021-04-14 06:26:44,538 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.250 2021-04-14 06:26:44,622 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.236 2021-04-14 06:26:44,622 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.236 2021-04-14 06:26:44,660 - root - INFO - Done with init on 172.23.106.250. 2021-04-14 06:26:44,687 - root - INFO - Done with cleanup on 172.23.106.250. 2021-04-14 06:26:44,740 - root - INFO - Done with init on 172.23.106.236. 2021-04-14 06:26:44,762 - root - INFO - Done with cleanup on 172.23.106.236. 2021-04-14 06:26:50,105 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.121.78 2021-04-14 06:26:50,107 - root - INFO - Setting KV memory quota as 256 MB on 172.23.121.78 2021-04-14 06:26:50,259 - root - INFO - Done with init on 172.23.121.78. 2021-04-14 06:26:50,282 - root - INFO - Done with cleanup on 172.23.121.78. 2021-04-14 06:27:53,740 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.105.175 2021-04-14 06:27:53,742 - root - INFO - Setting KV memory quota as 256 MB on 172.23.105.175 2021-04-14 06:27:53,865 - root - INFO - Done with init on 172.23.105.175. 2021-04-14 06:27:53,885 - root - INFO - Done with cleanup on 172.23.105.175. 2021-04-14 06:27:54,865 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.238 2021-04-14 06:27:54,865 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.238 2021-04-14 06:27:54,984 - root - INFO - Done with init on 172.23.106.238. 2021-04-14 06:27:55,006 - root - INFO - Done with cleanup on 172.23.106.238. 2021-04-14 06:27:55,467 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.233 2021-04-14 06:27:55,469 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.233 2021-04-14 06:27:55,631 - root - INFO - Done with init on 172.23.106.233. 2021-04-14 06:27:55,651 - root - INFO - Done with cleanup on 172.23.106.233. 2021-04-14 06:28:00,546 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.121.74 2021-04-14 06:28:00,548 - root - INFO - Setting KV memory quota as 256 MB on 172.23.121.74 2021-04-14 06:28:00,663 - root - INFO - Done with init on 172.23.121.74. 2021-04-14 06:28:00,687 - root - INFO - Done with cleanup on 172.23.121.74. 2021-04-14 06:28:17,296 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-04-14 06:28:17,309 - root - INFO - node:172.23.105.175:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,322 - root - INFO - node:172.23.106.233:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,335 - root - INFO - node:172.23.106.236:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,349 - root - INFO - node:172.23.106.238:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,365 - root - INFO - node:172.23.106.250:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,378 - root - INFO - node:172.23.106.251:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,391 - root - INFO - node:172.23.121.74:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,405 - root - INFO - node:172.23.121.78:8091 version:7.0.0-4907-enterprise aFamily:inet services:[u'kv'] 2021-04-14 06:28:17,407 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-04-14 06:28:17,407 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-04-14 06:28:17,407 - root - INFO - INSTALL COMPLETED ON: 172.23.105.175 2021-04-14 06:28:17,407 - root - INFO - INSTALL COMPLETED ON: 172.23.106.233 2021-04-14 06:28:17,407 - root - INFO - INSTALL COMPLETED ON: 172.23.106.236 2021-04-14 06:28:17,407 - root - INFO - INSTALL COMPLETED ON: 172.23.106.238 2021-04-14 06:28:17,408 - root - INFO - INSTALL COMPLETED ON: 172.23.106.250 2021-04-14 06:28:17,408 - root - INFO - INSTALL COMPLETED ON: 172.23.106.251 2021-04-14 06:28:17,408 - root - INFO - INSTALL COMPLETED ON: 172.23.121.74 2021-04-14 06:28:17,410 - root - INFO - INSTALL COMPLETED ON: 172.23.121.78 2021-04-14 06:28:17,410 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-04-14 06:28:17,414 - root - INFO - TOTAL INSTALL TIME = 511.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 8m 43s 1 actionable task: 1 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_for_volume_test,iterations=1,rerun=False,quota_percent=100,backup_service_test=True,services_for_rebalance_in=kv:backup,services_init=kv:backup-kv:backup-kv:backup-kv:backup -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_for_volume_test,iterations=1,rerun=False,quota_percent=100,backup_service_test=True,services_for_rebalance_in=kv:backup,services_init=kv:backup-kv:backup-kv:backup-kv:backup -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_for_volume_test,iterations=1,rerun=False,quota_percent=100,backup_service_test=True,services_for_rebalance_in=kv:backup,services_init=kv:backup-kv:backup-kv:backup-kv:backup -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_for_volume_test,iterations=1,rerun=False,quota_percent=100,backup_service_test=True,services_for_rebalance_in=kv:backup,services_init=kv:backup-kv:backup-kv:backup-kv:backup -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_for_volume_test,iterations=1,rerun=False,quota_percent=100,backup_service_test=True,services_for_rebalance_in=kv:backup,services_init=kv:backup-kv:backup-kv:backup-kv:backup -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_backup_service_volume/build/classes/java/main:/data/workspace/temp_backup_service_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_for_volume_test,iterations=1,rerun=False,quota_percent=100,backup_service_test=True,services_for_rebalance_in=kv:backup,services_init=kv:backup-kv:backup-kv:backup-kv:backup -m rest > Task :compileJava Note: /data/workspace/temp_backup_service_volume/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Global Test input params: {'cluster_name': 'durability_volume', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 8} Logs will be stored at /data/workspace/temp_backup_service_volume/logs/testrunner-21-Apr-14_06-28-25/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_for_volume_test,iterations=1,rerun=False,quota_percent=100,backup_service_test=True,services_for_rebalance_in=kv:backup,services_init=kv:backup-kv:backup-kv:backup-kv:backup' Test Input params: {'cluster_name': 'durability_volume', 'quota_percent': '100', 'ini': '/tmp/durability_volume.ini', 'iterations': '1', 'rerun': 'False', 'num_nodes': 8, 'logs_folder': '/data/workspace/temp_backup_service_volume/logs/testrunner-21-Apr-14_06-28-25/test_1', 'nodes_init': '4', 'services_for_rebalance_in': 'kv:backup', 'backup_service_test': 'True', 'services_init': 'kv:backup-kv:backup-kv:backup-kv:backup', 'bucket_spec': 'volume_templates.buckets_for_volume_test', 'case_number': 1, 'data_load_spec': 'volume_test_load_for_volume_test'} test_volume_taf (volumetests.Collections.volume) ... 2021-04-14 06:28:41,493 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= volume setup started for test #1 test_volume_taf ========= 2021-04-14 06:28:42,105 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_volume_taf ========= 2021-04-14 06:28:42,959 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0x7 type=client open_count=1 channel=[id: 0x54b4756e, 0.0.0.0/0.0.0.0:47082] timeout=300.0>) 2021-04-14 06:28:46,286 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0x8 type=client open_count=1 channel=[id: 0xaa43c1cb, 0.0.0.0/0.0.0.0:52381] timeout=300.0>) 2021-04-14 06:28:48,733 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.105.175:8091 during connect (<_realsocket at 0x9 type=client open_count=1 channel=[id: 0xe9071510, 0.0.0.0/0.0.0.0:37220] timeout=300.0>) 2021-04-14 06:28:50,753 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.105.175:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xa type=client open_count=1 channel=[id: 0xa4de6153, 0.0.0.0/0.0.0.0:60792] timeout=300.0>) 2021-04-14 06:28:56,140 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xb type=client open_count=1 channel=[id: 0x2239b778, 0.0.0.0/0.0.0.0:47106] timeout=300.0>) 2021-04-14 06:28:58,154 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0x622f4f56, 0.0.0.0/0.0.0.0:43208] timeout=300.0>) 2021-04-14 06:29:00,171 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.233:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xe1fe9f95, 0.0.0.0/0.0.0.0:58588] timeout=300.0>) 2021-04-14 06:29:02,187 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.233:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x8f7fab47, 0.0.0.0/0.0.0.0:36978] timeout=300.0>) 2021-04-14 06:29:07,536 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0xd7eda186, 0.0.0.0/0.0.0.0:35173] timeout=300.0>) 2021-04-14 06:29:09,549 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x83d55b7a, 0.0.0.0/0.0.0.0:51540] timeout=300.0>) 2021-04-14 06:29:11,565 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.236:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xca6ae4a4, 0.0.0.0/0.0.0.0:48134] timeout=300.0>) 2021-04-14 06:29:13,581 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.236:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x1fad2b01, 0.0.0.0/0.0.0.0:56481] timeout=300.0>) 2021-04-14 06:29:18,980 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x92ea3732, 0.0.0.0/0.0.0.0:54309] timeout=300.0>) 2021-04-14 06:29:20,997 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x32e45093, 0.0.0.0/0.0.0.0:49790] timeout=300.0>) 2021-04-14 06:29:23,015 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.238:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x6ccd92be, 0.0.0.0/0.0.0.0:55606] timeout=300.0>) 2021-04-14 06:29:25,029 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.238:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0xd0969fc8, 0.0.0.0/0.0.0.0:60268] timeout=300.0>) 2021-04-14 06:29:30,331 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x5cb8a550, 0.0.0.0/0.0.0.0:40178] timeout=300.0>) 2021-04-14 06:29:32,344 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x612c420a, 0.0.0.0/0.0.0.0:56897] timeout=300.0>) 2021-04-14 06:29:34,358 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.250:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0xf47aa915, 0.0.0.0/0.0.0.0:40326] timeout=300.0>) 2021-04-14 06:29:36,371 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.250:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x979603cd, 0.0.0.0/0.0.0.0:35205] timeout=300.0>) 2021-04-14 06:29:41,760 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x93ceed58, 0.0.0.0/0.0.0.0:36407] timeout=300.0>) 2021-04-14 06:29:43,773 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x87b86fba, 0.0.0.0/0.0.0.0:56787] timeout=300.0>) 2021-04-14 06:29:45,786 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.106.251:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0xde6d4a10, 0.0.0.0/0.0.0.0:54181] timeout=300.0>) 2021-04-14 06:29:47,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.251:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xb03369c2, 0.0.0.0/0.0.0.0:57753] timeout=300.0>) 2021-04-14 06:29:53,108 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0xa3a98b35, 0.0.0.0/0.0.0.0:55705] timeout=300.0>) 2021-04-14 06:29:55,121 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x370a451a, 0.0.0.0/0.0.0.0:45532] timeout=300.0>) 2021-04-14 06:29:57,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.74:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0x350e2d41, 0.0.0.0/0.0.0.0:44404] timeout=300.0>) 2021-04-14 06:29:59,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.74:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0xaaac2f08, 0.0.0.0/0.0.0.0:57983] timeout=300.0>) 2021-04-14 06:30:04,549 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0x16711b18, 0.0.0.0/0.0.0.0:40316] timeout=300.0>) 2021-04-14 06:30:06,562 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x3120b620, 0.0.0.0/0.0.0.0:36241] timeout=300.0>) 2021-04-14 06:30:08,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.78:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0xe2d568f7, 0.0.0.0/0.0.0.0:41262] timeout=300.0>) 2021-04-14 06:30:10,589 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.78:8091/nodes/self error [Errno 111] Connection refused 2021-04-14 06:30:49,115 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:591] RAM quota was defined less than 100 MB: 2021-04-14 06:31:03,102 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.105.175, 172.23.106.233, 172.23.106.236, 172.23.106.238, 172.23.106.250, 172.23.106.251, 172.23.121.74, 172.23.121.78 initialized 2021-04-14 06:31:03,115 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_volume_taf ========= 2021-04-14 06:31:03,131 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_volume_taf ========= 2021-04-14 06:31:12,171 | test | INFO | pool-3-thread-9 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 0.889724310777 | Cluster node | | 172.23.106.233 | ['kv,backup'] | | | <--- IN --- | | 172.23.106.236 | ['kv,backup'] | | | <--- IN --- | | 172.23.106.238 | ['kv,backup'] | | | <--- IN --- | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-14 06:31:12,183 | test | INFO | pool-3-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 06:31:22,220 | test | INFO | pool-3-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 06:31:22,234 | test | INFO | pool-3-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 10.0649998188 sec 2021-04-14 06:31:22,395 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-04-14 06:31:22,825 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-04-14 06:31:22,940 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 1 | none | 0 | 0 | 0 | 0 | 0 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1258291200 | 0 | 0 | | bucket3 | ephemeral | 1 | none | 0 | 0 | 0 | 0 | 0 | | default | couchbase | 1 | none | 0 | 0 | 0 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-04-14 06:31:22,941 | test | INFO | MainThread | [collections_base:collection_setup:162] Creating required SDK clients for client_pool 2021-04-14 06:31:42,119 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:31:42,140 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:33:49,778 | test | INFO | pool-3-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******........................*******..................................................*****X | 29.991 | | 10.003 | ........................*****X | 10.003 | | 10.000 | .......................X | 10.000 | | 9.998 | .......................******.*****X | 10.002 | | 10.000 | .......................X | 10.000 | | 9.999 | .......................*****........................................................*....................*****X | 40.004 | | 10.000 | .......................*****X | 10.000 | | 2.600 | .....X | 2.600 | | 0.000 | ******************..................*......***X | 9.999 | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:33:51,102 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******............................................................*******......................................************.*.*****X | 81952.658 | | 80351.767 | .................................................................................................***********X | 80682.228 | | 80139.107 | ................................................................................................******X | 80139.107 | | 78871.300 | ...............................................................................................******..*****X | 80514.300 | | 80368.700 | .................................................................................................X | 80368.700 | | 79908.206 | ................................................................................................************.****X | 80855.920 | +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 06:33:51,244 | test | INFO | pool-3-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *************...................................................................................................*****X | 19.994 | | 17.394 | .....................................................................................X | 17.394 | | 10.003 | .................................................******.................................................*****X | 19.996 | | 17.402 | ......................................................................................X | 17.402 | | 10.002 | .................................................*****....................................*.............*****X | 19.998 | | 10.001 | .................................................******..................................................******X | 20.000 | | 0.000 | ******..........................*.........................................................................****X | 19.999 | | 10.000 | .................................................*****X | 10.000 | | 9.999 | ................................................*****X | 9.999 | +--------+------------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:33:51,351 | test | INFO | pool-3-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******............*.....................................******..................................................*****X | 19.994 | | 0.000 | **************....................................................................................................*****X | 20.004 | | 17.400 | .....................................................................................X | 17.400 | | 12.398 | ............................................................*****..........................******X | 17.601 | | 10.000 | ................................................************..................................................************X | 20.001 | | 9.999 | ................................................*****X | 9.999 | +--------+-----------------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:33:51,802 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+------------+-----------------+-------------+-------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+------------+-----------------+-------------+-------------+------------------------+-------------------+-----------------------+ | 172.23.105.175 | backup, kv | 74.5227995758 | 25084055552 | 22467989504 | 990822400 / 3758092288 | 2327656 / 2359636 | 7.0.0-4907-enterprise | | 172.23.106.236 | backup, kv | 77.1844660194 | 25084067840 | 22757584896 | 600309760 / 3758092288 | 2326073 / 2311595 | 7.0.0-4907-enterprise | | 172.23.106.233 | backup, kv | 78.9294594948 | 25084071936 | 22703022080 | 0 / 3758092288 | 2326061 / 2351071 | 7.0.0-4907-enterprise | | 172.23.106.238 | backup, kv | 80.6329279456 | 25084071936 | 22742482944 | 46149632 / 3758092288 | 2318452 / 2276979 | 7.0.0-4907-enterprise | +----------------+------------+-----------------+-------------+-------------+------------------------+-------------------+-----------------------+ 2021-04-14 06:34:00,292 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-04-14 06:34:18,326 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4287] Bucket UID mismatch. Expected: 0, Actual: 1 2021-04-14 06:34:18,326 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4422] Bucket manifest UID mismatch! 2021-04-14 06:34:25,750 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4287] Bucket UID mismatch. Expected: 0, Actual: 1 2021-04-14 06:34:25,752 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4422] Bucket manifest UID mismatch! 2021-04-14 06:34:33,454 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4287] Bucket UID mismatch. Expected: 0, Actual: 1 2021-04-14 06:34:33,456 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4422] Bucket manifest UID mismatch! 2021-04-14 06:34:40,665 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4287] Bucket UID mismatch. Expected: 0, Actual: 1 2021-04-14 06:34:40,665 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4422] Bucket manifest UID mismatch! 2021-04-14 06:36:46,487 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | bucket1 | couchbase | 1 | none | 0 | 1500 | 419430400 | 78698960 | 48043532 | | bucket2 | couchbase | 2 | none | 0 | 1500 | 1258291200 | 108867616 | 74788958 | | bucket3 | ephemeral | 1 | none | 0 | 1500 | 419430400 | 55939088 | 136 | | default | couchbase | 1 | none | 0 | 10000000 | 71303168000 | 3997568720 | 4118077208 | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ 2021-04-14 06:36:46,489 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_volume_taf ========= 2021-04-14 06:36:51,260 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-04-14 06:36:57,835 | test | INFO | MainThread | [backup_service:create_predefined_plans:123] Created set(['_daily_backups', 'plan9', 'plan5', 'plan6', 'plan7', '_hourly_backups', 'plan8', 'plan1', 'plan2', 'plan10', 'plan3', 'plan4']) 2021-04-14 06:36:59,266 | test | INFO | MainThread | [backup_service:create_predefined_repos:136] Created set(['repo-plan6', 'repo-plan7', 'repo-plan4', 'repo-plan5', 'repo-plan8', 'repo-plan9', 'repo-plan10', 'repo-plan2', 'repo-plan3', 'repo-plan1']) 2021-04-14 06:36:59,267 | test | INFO | MainThread | [Collections:test_volume_taf:484] Finished steps 1-4 successfully in setup 2021-04-14 06:36:59,267 | test | INFO | MainThread | [Collections:test_volume_taf:490] Step 5: Rebalance in with Loading of docs 2021-04-14 06:36:59,318 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:36:59,328 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:37:03,236 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 5.87197580645 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 4.10475950642 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 4.53802639849 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 4.12254160363 | Cluster node | | 172.23.106.250 | ['kv,backup'] | | | <--- IN --- | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-14 06:37:03,260 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 06:37:08,349 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 5.68 2021-04-14 06:37:13,431 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:37:18,486 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 16.66 2021-04-14 06:37:23,555 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 21.21 2021-04-14 06:37:28,628 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 06:37:33,683 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.07 2021-04-14 06:37:38,740 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 37.72 2021-04-14 06:37:43,792 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 39.11 2021-04-14 06:37:48,851 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 40.33 2021-04-14 06:37:53,911 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.57 2021-04-14 06:37:58,977 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 42.49 2021-04-14 06:38:04,042 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 43.51 2021-04-14 06:38:09,091 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 44.26 2021-04-14 06:38:14,154 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 45.11 2021-04-14 06:38:19,210 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 45.98 2021-04-14 06:38:24,269 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 46.49 2021-04-14 06:38:29,322 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 47.11 2021-04-14 06:38:34,388 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 47.76 2021-04-14 06:38:39,444 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 48.39 2021-04-14 06:38:44,499 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 49.03 2021-04-14 06:38:47,924 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******.......................******.............................................................................X | 26.004 | | 26.004 | ..................................................................................................**.X | 26.004 | | 26.004 | ..................................................................................................*X | 26.004 | | 10.399 | ......................................*******.................*****X | 14.602 | | 8.800 | ................................******............................******X | 15.999 | | 14.801 | .......................................................*****X | 14.801 | | 13.600 | ...................................................*****X | 13.600 | | 4.800 | .................*******.................**X | 9.200 | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:38:48,384 | test | INFO | pool-3-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******.....................................................................................*****...........*...****X | 86767.417 | | 84955.270 | ................................................................................................X | 84955.270 | | 84342.870 | ................................................................................................*****X | 84342.870 | | 79973.486 | ...........................................................................................**X | 79973.486 | | 79973.486 | ...........................................................................................*********************X | 80517.396 | | 79745.100 | ..........................................................................................******.*****X | 80398.000 | | 78536.700 | .........................................................................................***X | 78536.700 | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 06:38:49,447 | test | INFO | pool-3-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******....................................................................................................*****X | 20.194 | | 8.602 | .........................................******.....................................*****X | 15.999 | | 12.402 | ............................................................******....................*******..*****X | 16.799 | | 15.801 | .............................................................................X | 15.801 | | 14.801 | ........................................................................****X | 14.801 | | 6.400 | ..............................*******................*****X | 9.600 | | 4.800 | ......................***X | 4.800 | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:38:49,540 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 49.66 2021-04-14 06:38:50,305 | test | INFO | pool-3-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******....................................................................................................*****X | 32.391 | | 16.402 | .................................................X | 16.402 | | 16.402 | .................................................****X | 16.402 | | 16.398 | .................................................*****X | 16.398 | | 6.001 | .................*******.....................*****X | 12.801 | | 10.800 | ................................******......*****X | 12.800 | | 0.000 | *******..................................*****X | 11.200 | | 4.800 | .............****X | 4.800 | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:38:59,584 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 06:38:59,608 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 116.371999979 sec 2021-04-14 06:39:10,160 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | bucket1 | couchbase | 1 | none | 0 | 1560 | 524288000 | 82172360 | 61807699 | | bucket2 | couchbase | 2 | none | 0 | 1560 | 1572864000 | 113033336 | 91784856 | | bucket3 | ephemeral | 1 | none | 0 | 1560 | 524288000 | 61569648 | 170 | | default | couchbase | 1 | none | 0 | 10400000 | 89128960000 | 4210256048 | 4534624691 | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ 2021-04-14 06:39:10,160 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 06:39:18,315 | test | INFO | MainThread | [Collections:test_volume_taf:499] Step 6: Rebalance Out with Loading of docs 2021-04-14 06:39:18,358 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:39:18,368 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:39:18,460 | test | INFO | pool-3-thread-12 | [table_view:display:72] Rebalance Overview +----------------+------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 12.0535150827 | Cluster node | | 172.23.106.250 | backup, kv | 7.0.0-4907-enterprise | 6.92006566486 | --- OUT ---> | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 14.7797551433 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 14.7718679103 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 18.2415361294 | Cluster node | +----------------+------------+-----------------------+---------------+--------------+ 2021-04-14 06:39:18,470 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 06:39:23,516 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:39:28,582 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:39:34,003 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:39:39,055 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 18.23 2021-04-14 06:39:44,108 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 23.15 2021-04-14 06:39:49,171 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 06:39:54,232 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 06:39:59,316 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 31.79 2021-04-14 06:40:04,384 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:40:09,434 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:40:14,484 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 37.79 2021-04-14 06:40:19,542 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 39.04 2021-04-14 06:40:24,609 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 40.21 2021-04-14 06:40:29,663 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 41.3 2021-04-14 06:40:34,720 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 42.1 2021-04-14 06:40:39,773 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 42.92 2021-04-14 06:40:44,819 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 43.51 2021-04-14 06:40:49,868 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 44.46 2021-04-14 06:40:54,924 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 45.21 2021-04-14 06:40:59,977 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 46.0 2021-04-14 06:41:05,036 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 46.91 2021-04-14 06:41:09,378 | test | INFO | pool-3-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *...................................................................................................**.X | 63.491 | | 63.491 | ..................................................................................................**X | 63.491 | | 38.400 | ...........................................................*****X | 38.400 | | 23.096 | ...................................X | 23.096 | | 18.600 | ............................*****X | 18.600 | | 5.000 | ......*****X | 5.000 | | 2.500 | ..******X | 2.500 | | 0.000 | ************...*****X | 2.501 | | 0.000 | ****************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:41:09,801 | test | INFO | pool-3-thread-19 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------------+------------+ | 0.000 | *..******.................................................................................................*****X | 100262.571 | | 85460.817 | ....................................................................................******...*.*****X | 89278.556 | | 85662.209 | ....................................................................................******...******X | 88640.200 | | 75211.973 | ..........................................................................********X | 75619.011 | | 75619.011 | ..........................................................................***...*****X | 78382.800 | | 77145.465 | ...........................................................................*****X | 77145.465 | | 74489.665 | .........................................................................****X | 74566.181 | +-----------+------------------------------------------------------------------------------------------------------------------+------------+ 2021-04-14 06:41:10,084 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 47.95 2021-04-14 06:41:10,778 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *....................................................................................................**X | 60.391 | | 60.391 | ...................................................................................................*X | 60.391 | | 60.391 | ...................................................................................................*X | 60.391 | | 35.299 | .........................................................*****X | 35.299 | | 7.000 | ..........*****X | 7.000 | | 6.200 | .........*************.....*****X | 9.500 | | 0.000 | *************....*****X | 2.500 | | 2.500 | ...*****X | 2.500 | | 0.000 | ****X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:41:11,071 | test | INFO | pool-3-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *.................******...................................................................................X | 59.299 | | 59.299 | ..................................................................................................****X | 59.299 | | 46.208 | ............................................................................X | 46.208 | | 10.901 | .................*****X | 10.901 | | 6.700 | ..........******.......*****X | 10.700 | | 10.500 | ................*.*****X | 11.100 | | 3.999 | .....******..*****X | 5.001 | | 5.000 | .......*****X | 5.000 | | 4.999 | .......******X | 4.999 | | 2.500 | ...****X | 2.500 | +--------+--------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:41:15,117 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 49.32 2021-04-14 06:41:20,150 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:41:25,181 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:41:30,213 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:41:40,253 | test | INFO | pool-3-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 06:41:40,298 | test | INFO | pool-3-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 141.838000059 sec 2021-04-14 06:41:48,802 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | bucket1 | couchbase | 1 | none | 0 | 1635 | 419430400 | 78343208 | 66749374 | | bucket2 | couchbase | 2 | none | 0 | 1635 | 1258291200 | 109005664 | 101433134 | | bucket3 | ephemeral | 1 | none | 0 | 1635 | 419430400 | 57220936 | 136 | | default | couchbase | 1 | none | 0 | 10816000 | 71303168000 | 4693006176 | 4820600506 | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ 2021-04-14 06:41:48,802 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 06:41:57,493 | test | INFO | MainThread | [Collections:test_volume_taf:508] Step 7: Rebalance In_Out with Loading of docs 2021-04-14 06:41:57,530 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:41:57,539 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:42:13,072 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 8.92113564669 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 12.9638980056 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 13.68156073 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 10.5926860025 | --- OUT ---> | | 172.23.106.251 | ['kv,backup'] | | | <--- IN --- | | 172.23.121.78 | ['kv,backup'] | | | <--- IN --- | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-14 06:42:13,088 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 06:42:18,154 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 5.36 2021-04-14 06:42:23,210 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 11.97 2021-04-14 06:42:28,256 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:42:33,316 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:42:38,387 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 14.14 2021-04-14 06:42:43,457 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 17.86 2021-04-14 06:42:48,528 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 21.44 2021-04-14 06:42:53,586 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 24.78 2021-04-14 06:42:58,641 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 06:43:03,698 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 06:43:08,789 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 29.84 2021-04-14 06:43:13,864 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-04-14 06:43:18,946 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:43:24,036 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:43:29,105 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 37.62 2021-04-14 06:43:34,190 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 38.35 2021-04-14 06:43:39,249 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 39.24 2021-04-14 06:43:44,299 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 40.13 2021-04-14 06:43:49,371 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 40.92 2021-04-14 06:43:54,457 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 41.71 2021-04-14 06:43:59,510 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 42.52 2021-04-14 06:44:02,391 | test | INFO | pool-3-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******..................................*..................................................................****X | 24.000 | | 20.800 | .....................................................................................******X | 20.800 | | 17.898 | .........................................................................*****X | 17.898 | | 6.501 | ..........................******....................*****X | 11.300 | | 10.600 | ...........................................X | 10.600 | | 8.599 | ..................................****X | 8.599 | | 5.200 | ....................X | 5.200 | | 3.800 | ..............******.........................................*X | 13.500 | | 12.900 | ....................................................***X | 12.900 | | 10.399 | ..........................................******X | 10.399 | | 4.700 | ..................******....................*****X | 9.400 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:44:02,706 | test | INFO | pool-3-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *......*****................*.................................................*****.........******...............*.....*****X | 93733.730 | | 78691.010 | ..................................................................................*****X | 78691.010 | | 76323.000 | ................................................................................******......X | 82285.264 | | 82285.264 | ......................................................................................****X | 82285.264 | | 80044.664 | ....................................................................................X | 80044.664 | | 72985.776 | ............................................................................*****.......*..*****X | 81002.551 | | 74447.930 | ..............................................................................*****X | 74447.930 | | 73222.730 | .............................................................................X | 73222.730 | | 71543.746 | ...........................................................................*****X | 71543.746 | | 70065.200 | .........................................................................*****X | 70065.200 | +-----------+-------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 06:44:03,872 | test | INFO | pool-3-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******....................*...............................................****X | 13.500 | | 7.800 | .....................................*******............................................................X | 19.798 | | 18.198 | .........................................................................................*****..........*****X | 20.101 | | 18.901 | .............................................................................................X | 18.901 | | 13.500 | ..................................................................*****X | 13.500 | | 8.600 | .........................................******.............*****X | 11.200 | | 11.000 | .....................................................*****X | 11.000 | | 6.200 | .............................X | 6.200 | | 4.700 | ......................******....................................................*****X | 15.101 | | 2.600 | ...........******..........................X | 7.800 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:44:04,565 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 43.52 2021-04-14 06:44:04,760 | test | INFO | pool-3-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******..*........******.................................................................................******.........*****X | 21.998 | | 18.602 | ...................................................................................X | 18.602 | | 17.502 | ..............................................................................****X | 17.502 | | 11.300 | ..................................................X | 11.300 | | 8.800 | .......................................******...............................X | 15.700 | | 15.201 | ....................................................................****X | 15.201 | | 14.200 | ...............................................................*****X | 14.200 | | 12.800 | .........................................................X | 12.800 | | 4.200 | ..................******.............................................*****X | 14.099 | | 9.900 | ............................................*****X | 9.900 | | 8.800 | .......................................X | 8.800 | | 5.200 | ......................*****X | 5.200 | | 2.600 | ..........X | 2.600 | +--------+-------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:44:09,605 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 44.4 2021-04-14 06:44:14,642 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 45.12 2021-04-14 06:44:19,681 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 45.98 2021-04-14 06:44:24,726 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 46.88 2021-04-14 06:44:29,766 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 47.51 2021-04-14 06:44:34,805 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 47.98 2021-04-14 06:44:39,842 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 48.24 2021-04-14 06:44:44,884 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 48.7 2021-04-14 06:44:49,923 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 48.85 2021-04-14 06:44:54,960 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 49.14 2021-04-14 06:45:00,000 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 49.49 2021-04-14 06:45:05,045 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 49.73 2021-04-14 06:45:10,084 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:45:15,124 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:45:20,165 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:45:25,200 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 64.29 2021-04-14 06:45:30,236 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 78.57 2021-04-14 06:45:40,279 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 06:45:40,328 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 207.257999897 sec 2021-04-14 06:45:50,565 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | bucket1 | couchbase | 1 | none | 0 | 1710 | 524288000 | 82127088 | 81415593 | | bucket2 | couchbase | 2 | none | 0 | 1710 | 1572864000 | 113578224 | 123478936 | | bucket3 | ephemeral | 1 | none | 0 | 1710 | 524288000 | 62268072 | 170 | | default | couchbase | 1 | none | 0 | 11248000 | 89128960000 | 5614369120 | 5700681058 | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ 2021-04-14 06:45:50,565 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 06:46:00,434 | test | INFO | MainThread | [Collections:test_volume_taf:517] Step 8: Swap with Loading of docs 2021-04-14 06:46:00,479 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:46:00,487 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:46:03,750 | test | INFO | pool-3-thread-20 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 8.87147731581 | Cluster node | | 172.23.121.78 | backup, kv | 7.0.0-4907-enterprise | 6.78094277792 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 10.3756933938 | Cluster node | | 172.23.106.251 | backup, kv | 7.0.0-4907-enterprise | 6.54229169293 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 10.5442176871 | --- OUT ---> | | 172.23.106.238 | ['kv,backup'] | | | <--- IN --- | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-14 06:46:03,770 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 06:46:08,834 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 7.53 2021-04-14 06:46:13,891 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:46:18,940 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:46:24,000 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:46:29,078 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 17.92 2021-04-14 06:46:34,131 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 24.11 2021-04-14 06:46:39,190 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 06:46:44,246 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 06:46:49,303 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 31.67 2021-04-14 06:46:54,384 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:46:59,453 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:47:04,516 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:47:09,572 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 38.15 2021-04-14 06:47:14,634 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 38.79 2021-04-14 06:47:19,683 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 39.81 2021-04-14 06:47:24,742 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.6 2021-04-14 06:47:29,802 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 41.38 2021-04-14 06:47:34,864 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 42.13 2021-04-14 06:47:39,934 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 42.87 2021-04-14 06:47:44,990 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 43.5 2021-04-14 06:47:50,039 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 44.11 2021-04-14 06:47:55,092 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 44.7 2021-04-14 06:47:55,357 | test | INFO | pool-3-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+-------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******................*...................................................................................X | 98.600 | | 98.600 | ..................................................................................................*X | 98.600 | | 98.600 | ..................................................................................................*X | 98.600 | | 98.600 | ..................................................................................................X | 98.600 | | 13.900 | .............*****X | 13.900 | | 7.100 | ......*******...*****X | 9.899 | | 0.000 | ******.*...****X | 4.900 | | 2.700 | .************X | 2.700 | | 0.000 | *******..*****X | 2.700 | | 0.000 | *X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:47:56,522 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******..............................................................******......................................X | 99272.600 | | 99272.600 | ..................................................................................................****X | 99272.600 | | 85992.700 | .....................................................................................*******......*****X | 92091.220 | | 85428.262 | .....................................................................................******.*....****X | 91121.600 | | 83558.995 | ...................................................................................******X | 83558.995 | | 82659.534 | ..................................................................................*******X | 83008.436 | | 83008.436 | ..................................................................................**X | 83008.436 | | 83008.436 | ..................................................................................X | 83008.436 | | 82800.100 | ..................................................................................*****X | 82800.100 | | 81388.672 | ................................................................................X | 81388.672 | | 81142.072 | ................................................................................*X | 81142.072 | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 06:47:57,099 | test | INFO | pool-3-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******....................................................................................................*****X | 60.200 | | 19.800 | ...............................X | 19.800 | | 12.200 | ...................*****X | 12.200 | | 7.300 | ...........******..................******X | 18.198 | | 7.300 | ...........******..*......****X | 12.100 | | 4.500 | ......X | 4.500 | | 2.700 | ...******...******.******X | 5.001 | | 2.200 | ..******....*..........*X | 10.299 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:47:57,799 | test | INFO | pool-3-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******...................................................................................................*****X | 97.600 | | 15.300 | ..............*****X | 15.300 | | 7.200 | ......******X | 7.200 | | 7.100 | ......*****X | 7.100 | | 2.200 | .*******...*****X | 4.900 | | 2.700 | .*****X | 2.700 | | 1.100 | X | 1.100 | | 0.000 | ******..*****X | 2.700 | | 2.700 | .*****X | 2.700 | | 1.100 | X | 1.100 | | 0.000 | **X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:48:00,130 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 45.41 2021-04-14 06:48:05,164 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 46.19 2021-04-14 06:48:10,197 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 46.96 2021-04-14 06:48:15,230 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 47.68 2021-04-14 06:48:20,262 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 48.03 2021-04-14 06:48:25,296 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 48.43 2021-04-14 06:48:30,329 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 48.7 2021-04-14 06:48:35,361 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 49.0 2021-04-14 06:48:40,394 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 49.29 2021-04-14 06:48:45,430 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 49.65 2021-04-14 06:48:50,463 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:48:55,497 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:49:00,533 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-14 06:49:05,568 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: running, progress: 85.71 2021-04-14 06:49:15,612 | test | INFO | pool-3-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 06:49:15,658 | test | INFO | pool-3-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 191.907999992 sec 2021-04-14 06:49:26,066 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ | bucket1 | couchbase | 1 | none | 0 | 1785 | 524288000 | 81954504 | 84899178 | | bucket2 | couchbase | 2 | none | 0 | 1785 | 1572864000 | 113828088 | 127620250 | | bucket3 | ephemeral | 1 | none | 0 | 1785 | 524288000 | 63125176 | 170 | | default | couchbase | 1 | none | 0 | 11698000 | 89128960000 | 5156276320 | 5775701034 | +---------+-----------+----------+------------+-----+----------+-------------+------------+------------+ 2021-04-14 06:49:26,068 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 06:49:36,572 | test | INFO | MainThread | [Collections:test_volume_taf:527] Step 9: Updating the bucket replica to 2 2021-04-14 06:49:36,786 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:49:36,792 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:49:40,296 | test | INFO | pool-3-thread-11 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 8.71102133569 | Cluster node | | 172.23.121.78 | backup, kv | 7.0.0-4907-enterprise | 6.9069826065 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 8.32913306452 | Cluster node | | 172.23.106.251 | backup, kv | 7.0.0-4907-enterprise | 3.67711875079 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 5.47564976028 | Cluster node | | 172.23.106.233 | ['kv,backup'] | | | <--- IN --- | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-14 06:49:40,322 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 06:49:45,423 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 2.1 2021-04-14 06:49:50,519 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 6.19 2021-04-14 06:49:55,586 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 10.03 2021-04-14 06:50:00,650 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 06:50:05,707 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 16.71 2021-04-14 06:50:10,796 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 23.26 2021-04-14 06:50:15,865 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 26.22 2021-04-14 06:50:20,938 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.99 2021-04-14 06:50:26,081 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 35.22 2021-04-14 06:50:31,161 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-04-14 06:50:36,244 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 37.82 2021-04-14 06:50:41,315 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 38.31 2021-04-14 06:50:46,380 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 38.8 2021-04-14 06:50:51,476 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.2 2021-04-14 06:50:56,556 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.52 2021-04-14 06:51:01,628 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.87 2021-04-14 06:51:06,698 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.17 2021-04-14 06:51:11,755 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.4 2021-04-14 06:51:16,816 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.7 2021-04-14 06:51:21,877 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 41.07 2021-04-14 06:51:26,957 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 41.42 2021-04-14 06:51:32,036 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 41.79 2021-04-14 06:51:37,098 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 42.06 2021-04-14 06:51:42,164 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 42.31 2021-04-14 06:51:47,213 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 42.57 2021-04-14 06:51:50,946 | test | INFO | pool-3-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **.......................................................................................*.............**X | 71.280 | | 71.280 | ..................................................................................................**X | 71.280 | | 19.805 | ..........................******..........*****X | 27.000 | | 4.700 | .....******..............*****X | 14.400 | | 0.000 | *******...........*****X | 7.900 | | 2.800 | ..*****X | 2.800 | | 0.000 | *****************************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:51:51,193 | test | INFO | pool-3-thread-10 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...............******..................................................................................******..*.*X | 90963.063 | | 90963.063 | ..................................................................................................*.X | 90963.063 | | 90963.063 | ..................................................................................................X | 90963.063 | | 87268.060 | ..............................................................................................*****X | 87268.060 | | 83567.327 | ..........................................................................................*****X | 83567.327 | | 83073.606 | ..........................................................................................******X | 83073.606 | | 71759.000 | .............................................................................****X | 71759.000 | | 71759.000 | .............................................................................X | 71759.000 | | 71104.052 | .............................................................................***********X | 71127.291 | | 68465.700 | ..........................................................................X | 68465.700 | | 68465.700 | ..........................................................................*********X | 68960.800 | | 68960.800 | ..........................................................................X | 68960.800 | | 67403.965 | .........................................................................**********X | 67925.028 | +-----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 06:51:52,267 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 42.91 2021-04-14 06:51:52,381 | test | INFO | pool-3-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **....................*...*****........................................................................*X | 19.605 | | 19.605 | ..............................................................................................*X | 19.605 | | 19.605 | ..............................................................................................**.....*****X | 20.500 | | 12.000 | .........................................................******...........*..*****X | 14.599 | | 14.301 | ....................................................................******....X | 15.101 | | 15.100 | ........................................................................****X | 15.100 | | 2.300 | ..........*******..............................................................*****X | 15.002 | | 13.000 | ..............................................................*****X | 13.000 | | 2.800 | ............******.......................*****X | 7.400 | | 5.100 | .......................*****X | 5.100 | +--------+-------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:51:52,726 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **...........*...*****......................................................................................******X | 50.909 | | 19.800 | .....................................*****X | 19.800 | | 9.499 | .................******.....*****X | 11.802 | | 7.400 | .............*****X | 7.400 | | 3.000 | ....X | 3.000 | | 2.300 | ...******...........................*****X | 15.899 | | 10.701 | ....................*****X | 10.701 | | 7.400 | .............*****X | 7.400 | | 0.000 | *******.....***********X | 2.800 | +--------+---------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:51:57,303 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 43.4 2021-04-14 06:52:02,339 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 43.96 2021-04-14 06:52:07,377 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 44.59 2021-04-14 06:52:12,417 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 45.15 2021-04-14 06:52:17,456 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 45.66 2021-04-14 06:52:22,490 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 46.09 2021-04-14 06:52:27,529 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 46.6 2021-04-14 06:52:32,569 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 47.07 2021-04-14 06:52:37,615 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 47.45 2021-04-14 06:52:42,655 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 47.78 2021-04-14 06:52:47,694 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 48.12 2021-04-14 06:52:52,734 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 48.54 2021-04-14 06:52:57,770 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 48.83 2021-04-14 06:53:02,805 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 49.25 2021-04-14 06:53:07,838 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 49.54 2021-04-14 06:53:12,871 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 49.76 2021-04-14 06:53:17,904 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 78.57 2021-04-14 06:53:23,148 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: running, progress: 92.86 2021-04-14 06:53:33,194 | test | INFO | pool-3-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 06:53:33,227 | test | INFO | pool-3-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 232.931999922 sec 2021-04-14 06:53:46,223 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | bucket1 | couchbase | 2 | none | 0 | 1860 | 629145600 | 118254352 | 118105174 | | bucket2 | couchbase | 2 | none | 0 | 1860 | 1887436800 | 119183056 | 130663330 | | bucket3 | ephemeral | 2 | none | 0 | 1860 | 629145600 | 92109376 | 204 | | default | couchbase | 2 | none | 0 | 12166000 | 106954752000 | 11959596928 | 10976801447 | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ 2021-04-14 06:53:46,223 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 06:54:01,073 | test | INFO | MainThread | [Collections:test_volume_taf:540] Enabling autoreprovison before inducing failure to prevent data loss for if there are ephemeral buckets 2021-04-14 06:54:01,105 | test | INFO | MainThread | [Collections:test_volume_taf:548] Step 10: stop_memcached 2021-04-14 06:54:01,107 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:54:01,114 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:54:01,740 | test | INFO | MainThread | [cb_error:create:39] Simulating 'stop_memcached' in 172.23.121.78 2021-04-14 06:54:01,842 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 20 seconds. Reason: Wait before reverting the error condition 2021-04-14 06:54:21,842 | test | INFO | MainThread | [cb_error:revert:68] Reverting 'stop_memcached' in 172.23.121.78 2021-04-14 06:55:37,428 | test | INFO | Thread-8574 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 06:55:37,437 | test | INFO | Thread-8574 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T06:55:31.038748187-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T06:55:30.962141527-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'd1bb1fcca1c1f2f0d02e38b219bb0d42', 'out': None, 'progress': 0.0, 'start': '2021-04-14T06:52:03.827807188-07:00', 'status': 'failed'}], 'start': '2021-04-14T06:52:03.744369647-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 06:56:22,841 | test | INFO | Thread-8576 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 06:56:22,845 | test | INFO | Thread-8576 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T06:56:17.84394146-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T06:56:17.789620436-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': '8bd71b8b14f6946498dc8f93a404582a', 'out': None, 'progress': 0.0, 'start': '2021-04-14T06:52:04.101674376-07:00', 'status': 'failed'}], 'start': '2021-04-14T06:52:03.954118107-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 06:56:49,394 | test | INFO | pool-3-thread-21 | [table_view:display:72] Rebalance Overview +----------------+------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 56.4322250639 | Cluster node | | 172.23.121.78 | backup, kv | 7.0.0-4907-enterprise | 21.488336714 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 33.099310697 | Cluster node | | 172.23.106.251 | backup, kv | 7.0.0-4907-enterprise | 13.2488917036 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 16.5528578655 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 18.0164018452 | Cluster node | +----------------+------------+-----------------------+---------------+--------------+ 2021-04-14 06:56:49,414 | test | INFO | pool-3-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 06:56:49,954 | test | INFO | pool-3-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.....******X | 2.200 | | 0.200 | *****..................................................................................*..................*****X | 42.007 | | 23.200 | ......................................................****X | 23.200 | | 14.399 | .................................X | 14.399 | | 12.599 | ............................****X | 12.599 | | 10.199 | .......................X | 10.199 | | 5.601 | ............*****.*.....****X | 8.200 | | 7.800 | .................******.......*****X | 10.601 | | 5.399 | ...........******....................*****X | 13.803 | | 5.000 | ..........****X | 5.000 | | 4.499 | .........*..****X | 5.401 | | 0.700 | X | 0.700 | | 0.000 | ******...................*****X | 8.400 | | 2.400 | ....***X | 2.400 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:56:50,132 | test | INFO | pool-3-thread-22 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **********X | 135.200 | | 44.492 | X | 44.492 | | 14.892 | *****..................................................................................*X | 67210.154 | | 67210.154 | .................................................................................****..................*****X | 81429.189 | | 75937.032 | ............................................................................................*****X | 75937.032 | | 71717.443 | .......................................................................................*****X | 71717.443 | | 71029.000 | ......................................................................................****X | 71029.000 | | 68971.039 | ...................................................................................X | 68971.039 | | 68507.539 | ...................................................................................******....****X | 71697.017 | | 70791.449 | .....................................................................................X | 70791.449 | | 66901.690 | .................................................................................*****....****X | 70258.896 | | 63688.573 | .............................................................................X | 63688.573 | | 62257.171 | ...........................................................................******..******.*****.........*****X | 72065.400 | | 67644.000 | ..................................................................................****X | 67644.000 | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 06:56:51,397 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****************...................................................................................................*****X | 47.308 | | 15.600 | ...............................*****X | 15.600 | | 5.199 | .........******...........*****X | 10.401 | | 7.600 | ...............******..........*****X | 12.599 | | 5.000 | .........*****......*****X | 7.997 | | 7.698 | ...............X | 7.698 | | 5.402 | ..........*****......*****X | 8.398 | | 3.001 | .....******......................*****X | 13.600 | | 8.400 | ................*****X | 8.400 | | 0.000 | ****X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:56:52,282 | test | INFO | pool-3-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.............*****X | 8.300 | | 3.249 | ....X | 3.249 | | 1.049 | *****...................................................................................................*****X | 63.011 | | 8.200 | ............******...........****X | 15.398 | | 5.000 | ......*****X | 5.000 | | 5.000 | ......******.....******....****X | 10.400 | | 5.699 | ........X | 5.699 | | 4.999 | ......****X | 4.999 | | 4.499 | ......*......****X | 8.402 | | 5.399 | .......******......*****X | 9.002 | | 0.000 | ******.........*****X | 6.000 | | 0.000 | *****X | 0.000 | +-------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 06:56:59,494 | test | INFO | pool-3-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 06:56:59,526 | test | INFO | pool-3-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 10.1319999695 sec 2021-04-14 06:57:12,622 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+--------------+------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+--------------+------------+-------------+ | bucket1 | couchbase | 2 | none | 0 | 1950 | 629145600 | 117745488 | 132832645 | | bucket2 | couchbase | 2 | none | 0 | 1950 | 1887436800 | 118546256 | 145247296 | | bucket3 | ephemeral | 2 | none | 0 | 1950 | 629145600 | 90155040 | 204 | | default | couchbase | 2 | none | 0 | 12652000 | 106954752000 | 8217990992 | 14121653429 | +---------+-----------+----------+------------+-----+----------+--------------+------------+-------------+ 2021-04-14 06:57:12,624 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 06:57:18,750 | test | INFO | Thread-8578 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 06:57:27,193 | test | INFO | MainThread | [Collections:test_volume_taf:548] Step 11: stop_prometheus 2021-04-14 06:57:27,194 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 06:57:27,200 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 06:57:27,838 | test | INFO | MainThread | [cb_error:create:39] Simulating 'stop_prometheus' in 172.23.121.78 2021-04-14 06:57:27,940 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 20 seconds. Reason: Wait before reverting the error condition 2021-04-14 06:57:47,943 | test | INFO | MainThread | [cb_error:revert:68] Reverting 'stop_prometheus' in 172.23.121.78 2021-04-14 06:58:55,028 | test | INFO | Thread-9650 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 06:58:55,036 | test | INFO | Thread-9650 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T06:58:53.480688204-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T06:58:53.324084113-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'd1bb1fcca1c1f2f0d02e38b219bb0d42', 'out': None, 'progress': 0.0, 'start': '2021-04-14T06:55:30.974735234-07:00', 'status': 'failed'}], 'start': '2021-04-14T06:52:03.873417552-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:00:01,670 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +----------------+------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 73.6897137629 | Cluster node | | 172.23.121.78 | backup, kv | 7.0.0-4907-enterprise | 24.387755102 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 83.3829429274 | Cluster node | | 172.23.106.251 | backup, kv | 7.0.0-4907-enterprise | 69.711857655 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 74.5244654336 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 84.0150142376 | Cluster node | +----------------+------------+-----------------------+---------------+--------------+ 2021-04-14 07:00:01,697 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 07:00:03,551 | test | INFO | pool-3-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *....*......................................................................................*..........X | 45.986 | | 21.738 | ..............................................X | 21.738 | | 21.738 | ..............................................*X | 21.738 | | 21.738 | ..............................................**X | 21.738 | | 16.500 | ..................................*****X | 16.500 | | 8.897 | ..................*****X | 8.897 | | 8.100 | ................*****X | 8.100 | | 2.700 | ....******...............................*****X | 16.699 | | 3.100 | .....******...*.........****X | 8.397 | | 5.702 | ...........******......*****X | 8.299 | | 3.100 | .....****X | 3.100 | | 3.099 | .....******.............****X | 8.803 | | 1.103 | .X | 1.103 | +--------+----------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:00:03,628 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.........*...............................................................................*............X | 91018.023 | | 81340.526 | ........................................................................................*****X | 81818.900 | | 81550.300 | ........................................................................................*****..*.****X | 83839.288 | | 74546.574 | ................................................................................******.*..*****.*.....****X | 82005.247 | | 65634.928 | .......................................................................******X | 66146.551 | | 66146.551 | .......................................................................***X | 66146.551 | | 61223.890 | ..................................................................******...****X | 64012.187 | | 61498.487 | ..................................................................X | 61498.487 | | 60938.000 | .................................................................****X | 60938.000 | | 60054.524 | ................................................................X | 60054.524 | | 55363.124 | ...........................................................****X | 55363.124 | | 54210.623 | ..........................................................*****...X | 56715.008 | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 07:00:05,114 | test | INFO | pool-3-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *....*....................................................................................*............X | 44.269 | | 20.003 | ............................................*****X | 20.003 | | 10.700 | .......................******.....................................*****X | 27.091 | | 5.401 | ...........******.............*****X | 11.401 | | 2.700 | .....******........******....**********X | 8.303 | | 0.000 | ******.............****X | 5.800 | | 5.600 | ...........X | 5.600 | | 3.100 | ......****X | 3.100 | | 0.000 | ******............X | 5.699 | +--------+----------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:00:05,657 | test | INFO | pool-3-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *............................*............................................................*............X | 40.586 | | 23.934 | .........................................................****X | 23.934 | | 9.000 | .....................X | 9.000 | | 5.400 | ............*****...............*....****X | 13.297 | | 2.601 | .....******.******.......................................*****X | 18.999 | | 8.401 | ...................*****X | 8.401 | | 3.099 | ......*****.....*..*****.......**********X | 8.800 | | 6.000 | .............X | 6.000 | | 5.698 | .............****X | 5.698 | | 0.000 | ******.......*X | 3.100 | +--------+----------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:00:11,842 | test | INFO | pool-3-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 07:00:11,911 | test | INFO | pool-3-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2409999371 sec 2021-04-14 07:00:28,150 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+--------------+------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+--------------+------------+-------------+ | bucket1 | couchbase | 2 | none | 0 | 2025 | 629145600 | 117109344 | 148238013 | | bucket2 | couchbase | 2 | none | 0 | 2025 | 1887436800 | 117052640 | 160630051 | | bucket3 | ephemeral | 2 | none | 0 | 2025 | 629145600 | 89561488 | 204 | | default | couchbase | 2 | none | 0 | 13158000 | 106954752000 | 8393516664 | 14879383264 | +---------+-----------+----------+------------+-----+----------+--------------+------------+-------------+ 2021-04-14 07:00:28,151 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 07:00:42,926 | test | INFO | MainThread | [Collections:test_volume_taf:567] Step 12: Graceful Failover a node and RebalanceOut that node with data load in parallel 2021-04-14 07:00:43,016 | test | INFO | MainThread | [Collections:test_volume_taf:576] Collecting pre_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 07:01:34,099 | test | INFO | Thread-9657 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:01:34,101 | test | INFO | Thread-9657 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:01:57,263 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 07:01:57,273 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 07:03:41,858 | test | INFO | Thread-10731 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:03:41,861 | test | INFO | Thread-10731 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:03:28.301537659-07:00', 'error': 'exit status 1: ', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:03:28.228879444-07:00', 'error': 'exit status 1: ', 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 75.17541737236874, 'start': '2021-04-14T07:01:18.256985895-07:00', 'status': 'failed'}], 'start': '2021-04-14T06:52:03.628997803-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:03:41,861 | test | INFO | Thread-10731 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:03:41,864 | test | INFO | Thread-10731 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:03:37.848884809-07:00', 'error': 'exit status 1: ', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:03:37.778651249-07:00', 'error': 'exit status 1: ', 'error_code': 2, 'node_id': '8bd71b8b14f6946498dc8f93a404582a', 'out': None, 'progress': 50.20566174691508, 'start': '2021-04-14T06:56:17.794311188-07:00', 'status': 'failed'}], 'start': '2021-04-14T06:52:03.994508776-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:04:40,121 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **.............*****.......................................................................................*****X | 42.193 | | 38.804 | ..........................................................................................X | 38.804 | | 25.404 | ...........................................................****X | 25.404 | | 3.199 | ......******................................*****X | 16.502 | | 14.799 | ..................................******X | 14.799 | | 11.900 | ...........................*****X | 11.900 | | 2.800 | .....******......................*****X | 11.900 | | 2.800 | .....******...............*****X | 9.101 | | 5.900 | ............******.*****X | 6.000 | | 5.500 | ............*****X | 5.500 | | 5.400 | ...........*.*****X | 5.900 | | 3.200 | ......***X | 3.200 | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:04:40,944 | test | INFO | pool-3-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........*****...............*............................................................................*.X | 88022.609 | | 88022.609 | ..................................................................................................*.*X | 88022.609 | | 71449.886 | ................................................................................*X | 71449.886 | | 68649.986 | ............................................................................***X | 68649.986 | | 55796.190 | ..............................................................*****.......................*X | 75706.311 | | 75706.311 | .....................................................................................**X | 75706.311 | | 75706.311 | .....................................................................................*....*****X | 79987.838 | | 72668.885 | .................................................................................***X | 72668.885 | | 72668.885 | .................................................................................*X | 72668.885 | | 63571.135 | .......................................................................******........*****X | 70653.200 | | 67655.237 | ...........................................................................******......*****X | 72623.767 | | 64512.372 | ........................................................................******........******....*****X | 75437.400 | | 68478.700 | ............................................................................*****X | 68478.700 | | 67363.200 | ...........................................................................****X | 67363.200 | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 07:04:41,266 | test | INFO | pool-3-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******....................................................................................................*****X | 28.295 | | 14.002 | ................................................*****X | 14.002 | | 2.799 | ........******..........******..........*****X | 8.300 | | 2.800 | ........******....................*****X | 8.301 | | 5.400 | ..................******...............................*****X | 14.199 | | 11.201 | ......................................******...******...........*****X | 15.101 | | 11.300 | ......................................******..*****X | 11.800 | | 8.600 | .............................****X | 8.600 | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:04:41,480 | test | INFO | pool-3-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******....................................................................................................*****X | 28.195 | | 22.204 | .............................................................................*****X | 22.204 | | 14.197 | .................................................*****X | 14.197 | | 5.901 | ...................******..........................................*****X | 17.499 | | 14.000 | ................................................*****X | 14.000 | | 2.800 | ........******.......................******.............................*****X | 17.299 | | 11.501 | .......................................*****X | 11.501 | | 0.000 | *******....................******..........******.............*****X | 12.300 | | 3.200 | ..........****X | 3.200 | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:05:53,052 | test | INFO | Thread-10737 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:05:53,055 | test | INFO | Thread-10737 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:05:52.528568352-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket1\': failed to transfer bucket data for bucket \'bucket1\': failed to transfer key value data: failed to transfer key value data: failed to initialise worker 3: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket1","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"61bb7970-4fe1-4d59-8088-7ae920b2daae","last_dispatched_to":"172.23.121.78:11210","last_dispatched_from":"172.23.106.238:45216","last_connection_id":"d088b49ef7a4894c/079a126663e504b4"}', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:05:52.498457871-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket1\': failed to transfer bucket data for bucket \'bucket1\': failed to transfer key value data: failed to transfer key value data: failed to initialise worker 3: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket1","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"61bb7970-4fe1-4d59-8088-7ae920b2daae","last_dispatched_to":"172.23.121.78:11210","last_dispatched_from":"172.23.106.238:45216","last_connection_id":"d088b49ef7a4894c/079a126663e504b4"}', 'error_code': 2, 'node_id': '30ff20027c87bcb7260bc9f7046b6daa', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:01:29.180923304-07:00', 'status': 'failed'}], 'start': '2021-04-14T06:52:03.779083436-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:06:57,540 | test | INFO | MainThread | [Collections:wait_for_failover_or_assert:459] 1 nodes failed over as expected in 0.039999961853 seconds 2021-04-14 07:07:21,096 | test | INFO | Thread-10741 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:07:21,099 | test | INFO | Thread-10741 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:07:20.271219444-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:07:20.207335248-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': '8bd71b8b14f6946498dc8f93a404582a', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:03:37.783317337-07:00', 'status': 'failed'}], 'start': '2021-04-14T06:52:04.05841039-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:07:21,099 | test | INFO | Thread-10741 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Allowed) 2021-04-14 07:07:21,101 | test | INFO | Thread-10741 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:07:21,102 | test | INFO | Thread-10741 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:07:04.587193741-07:00', 'error': 'exit status 1: failed to lock archive: failed to sync lockfile to disk: sync /tmp/my_archive/archive-plan9/lock.lk: input/output error', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:07:04.508907817-07:00', 'error': 'exit status 1: failed to lock archive: failed to sync lockfile to disk: sync /tmp/my_archive/archive-plan9/lock.lk: input/output error', 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:07:04.433845503-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:07:04.301507701-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:08:45,301 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 107.632000208 seconds to complete 2021-04-14 07:08:45,305 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-04-14 07:09:11,368 | test | INFO | MainThread | [Collections:test_volume_taf:638] Collecting post_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root] 2021-04-14 07:09:12,092 | test | INFO | Thread-10745 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:09:12,096 | test | INFO | Thread-10745 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:08:51.778024443-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:08:51.746733446-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': '8bd71b8b14f6946498dc8f93a404582a', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:07:20.210095181-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:07:04.204058868-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:09:59,029 | test | INFO | Thread-10747 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:09:59,032 | test | INFO | Thread-10747 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:09:52.252743114-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:09:52.202832696-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:07:04.737188024-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:07:04.637510351-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:11:23,279 | test | INFO | Thread-10751 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:11:44,730 | test | INFO | Thread-10752 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:11:44,732 | test | INFO | Thread-10752 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:11:28.396103503-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:11:28.359791845-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': '30ff20027c87bcb7260bc9f7046b6daa', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:07:04.822717511-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:07:04.72291963-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:12:07,141 | test | INFO | Thread-10753 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:12:31,210 | test | INFO | pool-3-thread-19 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 18.6763952893 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 9.91923271075 | Cluster node | | 172.23.106.251 | backup, kv | 7.0.0-4907-enterprise | 3.86119873817 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 8.15154994259 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 4.18404752876 | Cluster node | | 172.23.121.74 | ['kv,backup'] | | | <--- IN --- | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-14 07:12:31,219 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 07:12:36,292 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 1.49 2021-04-14 07:12:41,339 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 8.74 2021-04-14 07:12:46,382 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 07:12:51,436 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 19.56 2021-04-14 07:12:56,492 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-04-14 07:13:01,559 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 29.2 2021-04-14 07:13:06,612 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 36.4 2021-04-14 07:13:11,700 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 37.57 2021-04-14 07:13:16,753 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 38.06 2021-04-14 07:13:21,799 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 38.55 2021-04-14 07:13:26,849 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 39.06 2021-04-14 07:13:31,914 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 39.48 2021-04-14 07:13:36,970 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 40.06 2021-04-14 07:13:42,016 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 40.55 2021-04-14 07:13:47,061 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 40.97 2021-04-14 07:13:52,108 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 41.48 2021-04-14 07:13:57,147 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 41.85 2021-04-14 07:14:01,772 | test | INFO | Thread-10758 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:14:01,773 | test | INFO | Thread-10758 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:13:51.585896718-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:13:51.540786756-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'd54bc8af7e37630cd7d6f8e576460a3b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:11:10.604993626-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:07:04.58684138-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:14:02,190 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 42.31 2021-04-14 07:14:07,242 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 42.82 2021-04-14 07:14:12,279 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 43.21 2021-04-14 07:14:17,319 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 43.73 2021-04-14 07:14:22,249 | test | INFO | Thread-10759 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:14:22,358 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 44.22 2021-04-14 07:14:27,400 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 44.7 2021-04-14 07:14:32,441 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 45.14 2021-04-14 07:14:37,483 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 45.68 2021-04-14 07:14:42,526 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 46.14 2021-04-14 07:14:47,575 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 46.61 2021-04-14 07:14:52,627 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 47.17 2021-04-14 07:14:57,668 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 47.49 2021-04-14 07:15:02,717 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 48.05 2021-04-14 07:15:07,773 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 48.46 2021-04-14 07:15:12,811 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 48.88 2021-04-14 07:15:17,865 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 49.2 2021-04-14 07:15:22,947 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 49.54 2021-04-14 07:15:28,003 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 49.9 2021-04-14 07:15:33,085 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: running, progress: 85.71 2021-04-14 07:15:43,157 | test | INFO | pool-3-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 07:15:43,194 | test | INFO | pool-3-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 191.983999968 sec 2021-04-14 07:15:43,375 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | bucket1 | couchbase | 2 | none | 0 | 2100 | 629145600 | 118489856 | 173105423 | | bucket2 | couchbase | 2 | none | 0 | 2100 | 1887436800 | 118781280 | 188153667 | | bucket3 | ephemeral | 2 | none | 0 | 2100 | 629145600 | 90349984 | 204 | | default | couchbase | 2 | none | 0 | 13685000 | 106954752000 | 10968682712 | 11828270445 | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ 2021-04-14 07:15:43,375 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 07:16:01,061 | test | INFO | MainThread | [Collections:test_volume_taf:567] Step 13: Graceful Failover a node and FullRecovery that node with data load in parallel 2021-04-14 07:16:01,142 | test | INFO | MainThread | [Collections:test_volume_taf:576] Collecting pre_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root] 2021-04-14 07:17:11,582 | test | INFO | Thread-10766 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:17:11,930 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 07:17:11,931 | test | INFO | Thread-10766 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:17:05.23554017-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:17:05.161662022-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_id': '30ff20027c87bcb7260bc9f7046b6daa', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:05.064621019-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:04.948058638-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:17:11,931 | test | INFO | Thread-10766 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:17:11,934 | test | INFO | Thread-10766 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:17:11.272724198-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:17:11.087102478-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:04.987388441-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:04.894038592-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:17:11,934 | test | INFO | Thread-10766 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:17:11,936 | test | INFO | Thread-10766 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:17:11.22686655-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:17:05.395137211-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_id': 'd54bc8af7e37630cd7d6f8e576460a3b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:05.320837913-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:05.238132042-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:17:11,937 | test | INFO | Thread-10766 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:17:11,938 | test | INFO | Thread-10766 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:16:52.962888912-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:16:52.927370098-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:14:10.291060528-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:07:04.512830824-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:17:11,938 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 07:17:44,017 | test | INFO | Thread-10767 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:17:44,026 | test | INFO | Thread-10767 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:17:15.702594285-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:17:15.478797295-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_id': '8bd71b8b14f6946498dc8f93a404582a', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:04.962010044-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:04.847624536-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:17:44,026 | test | INFO | Thread-10767 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:17:44,032 | test | INFO | Thread-10767 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:17:11.349730216-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:17:11.13948776-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:11.095869784-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:04.985493295-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:17:44,032 | test | INFO | Thread-10767 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:17:44,033 | test | INFO | Thread-10767 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:17:11.434766505-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:17:11.196863697-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:11.142762154-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:05.06649869-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:18:57,246 | test | INFO | Thread-11838 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:18:57,260 | test | INFO | Thread-11838 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:18:52.096192837-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket1': failed to transfer bucket data for bucket 'bucket1': failed to transfer key value data: failed to transfer key value data: failed to perform full scan: failed to perform full scan on chunk 0 for vBucket 255: failed to determine the offset of the first requested sequence number 0: failed to query offset for sequence number: 0: database disk image is malformed", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:18:52.033227438-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket1': failed to transfer bucket data for bucket 'bucket1': failed to transfer key value data: failed to transfer key value data: failed to perform full scan: failed to perform full scan on chunk 0 for vBucket 255: failed to determine the offset of the first requested sequence number 0: failed to query offset for sequence number: 0: database disk image is malformed", 'error_code': 2, 'node_id': 'ecf2dc24125339004ba632ffc3d05508', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:05.380863411-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:05.292637947-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:18:57,260 | test | INFO | Thread-11838 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:18:57,263 | test | INFO | Thread-11838 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:18:44.789768017-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket2': failed to transfer bucket data for bucket 'bucket2': failed to transfer key value data: failed to transfer key value data: failed to open vBucket 124: failed to open index: invalid Rift index: unknown index version; maybe the index is corrupt?", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:18:44.706454432-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket2': failed to transfer bucket data for bucket 'bucket2': failed to transfer key value data: failed to transfer key value data: failed to open vBucket 124: failed to open index: invalid Rift index: unknown index version; maybe the index is corrupt?", 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:17:04.886118527-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:04.808923715-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:18:57,263 | test | INFO | Thread-11838 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:18:57,266 | test | INFO | Thread-11838 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:18:45.977228958-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:18:45.875513299-07:00', 'error': 'exit status 1: failed to filter backups: backup range resulted in a single backup', 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:18:44.711134053-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:05.136372344-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:18:57,267 | test | INFO | Thread-11838 | [backup_service:monitor:87] The backup service task task1 has completed with status: failed. (Disallowed) 2021-04-14 07:18:57,270 | test | INFO | Thread-11838 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:18:46.143952422-07:00', 'error': "exit status 1: failed to lock archive; an old style lockfile still exists, if you are certain there isn't another cbbackupmgr process running remove '/tmp/my_archive/archive-plan9/lock.lk'", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:18:45.928105563-07:00', 'error': "exit status 1: failed to lock archive; an old style lockfile still exists, if you are certain there isn't another cbbackupmgr process running remove '/tmp/my_archive/archive-plan9/lock.lk'", 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:18:45.880456022-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:17:05.171788208-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task1', 'type': 'MERGE'} 2021-04-14 07:19:32,801 | test | INFO | pool-3-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****....................................................................................................*****X | 82.000 | | 15.200 | .................*****X | 15.200 | | 14.198 | ................*****X | 14.198 | | 5.298 | .....X | 5.298 | | 3.300 | ...******......************.X | 9.401 | | 9.001 | .........*****X | 9.001 | | 6.100 | ......*******.******..******.******X | 9.400 | | 0.000 | *******....*****X | 3.300 | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:19:33,944 | test | INFO | pool-3-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****.........*.............................................*****......................................******..X | 89811.436 | | 88332.936 | ...........................................................................................*X | 88332.936 | | 85794.236 | ........................................................................................***X | 85794.236 | | 69430.538 | .......................................................................******............................*****X | 95451.037 | | 92360.113 | ...............................................................................................X | 92360.113 | | 91581.013 | ..............................................................................................*X | 91581.013 | | 91581.013 | ..............................................................................................***X | 91581.013 | | 89253.748 | ............................................................................................*****X | 89253.748 | | 89059.077 | ............................................................................................X | 89059.077 | | 88958.177 | ............................................................................................*****X | 88958.177 | | 85638.595 | ........................................................................................*****X | 85638.595 | | 85036.895 | ........................................................................................X | 85036.895 | | 82967.600 | .....................................................................................******.******.******X | 84159.000 | | 81803.400 | ....................................................................................******.X | 83005.800 | +-----------+-----------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 07:19:34,046 | test | INFO | pool-3-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****..........................*..........................................................................****X | 44.500 | | 23.400 | ...................................................******.......**X | 26.696 | | 26.696 | ..........................................................**X | 26.696 | | 26.696 | ..........................................................X | 26.696 | | 2.900 | .....******..................................*****X | 18.002 | | 0.000 | ******...................*.......******.............*****X | 17.499 | | 6.600 | .............X | 6.600 | | 2.800 | .....*****X | 2.800 | | 0.000 | ******...................................******X | 15.600 | | 9.400 | ....................*****X | 9.400 | | 2.300 | ....X | 2.300 | | 0.000 | ******..............X | 6.600 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:19:34,414 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****....................................................................................................*****X | 41.300 | | 14.400 | .................................******.......*******................*****X | 23.799 | | 5.801 | .............*****X | 5.801 | | 0.000 | ******.......*.............................*****X | 15.101 | | 11.799 | ...........................*****X | 11.799 | | 6.101 | .............*******..............******.................******X | 18.800 | | 0.000 | ******..............*******.X | 6.600 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:22:12,227 | test | INFO | MainThread | [Collections:wait_for_failover_or_assert:459] 1 nodes failed over as expected in 0.0429999828339 seconds 2021-04-14 07:22:12,427 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +----------------+------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+------------+-----------------------+----------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 9.29968454259 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 0.476130810675 | Cluster node | | 172.23.106.251 | backup, kv | 7.0.0-4907-enterprise | 2.90456431535 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 2.99220517978 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 2.74593777554 | Cluster node | | 172.23.121.74 | backup, kv | 7.0.0-4907-enterprise | 3.08098591549 | Cluster node | +----------------+------------+-----------------------+----------------+--------------+ 2021-04-14 07:22:12,437 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 07:22:17,476 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 1.56 2021-04-14 07:22:22,532 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 12.5 2021-04-14 07:22:27,582 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 18.03 2021-04-14 07:22:32,625 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.32 2021-04-14 07:22:37,671 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 35.28 2021-04-14 07:22:42,720 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 37.52 2021-04-14 07:22:47,763 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 37.91 2021-04-14 07:22:52,802 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 38.43 2021-04-14 07:22:57,844 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 38.79 2021-04-14 07:23:02,884 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 39.25 2021-04-14 07:23:07,941 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 39.57 2021-04-14 07:23:12,990 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 39.98 2021-04-14 07:23:18,029 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 40.35 2021-04-14 07:23:23,071 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 40.79 2021-04-14 07:23:28,114 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.27 2021-04-14 07:23:29,013 | test | INFO | Thread-11850 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:23:29,016 | test | INFO | Thread-11850 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:23:11.822232992-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket1': failed to transfer bucket data for bucket 'bucket1': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:23:11.783136282-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket1': failed to transfer bucket data for bucket 'bucket1': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:22:05.671024017-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:22:05.573005705-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:23:33,154 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.61 2021-04-14 07:23:38,196 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 42.0 2021-04-14 07:23:43,234 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 42.42 2021-04-14 07:23:48,273 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 42.83 2021-04-14 07:23:53,339 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 43.32 2021-04-14 07:23:58,398 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 43.75 2021-04-14 07:24:03,437 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 44.23 2021-04-14 07:24:08,476 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 44.72 2021-04-14 07:24:13,516 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 45.18 2021-04-14 07:24:18,559 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 45.56 2021-04-14 07:24:23,605 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 46.1 2021-04-14 07:24:28,648 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 46.61 2021-04-14 07:24:33,700 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 47.07 2021-04-14 07:24:38,750 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 47.54 2021-04-14 07:24:43,786 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 47.93 2021-04-14 07:24:48,828 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 48.39 2021-04-14 07:24:53,871 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 48.88 2021-04-14 07:24:58,915 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 49.32 2021-04-14 07:25:01,099 | test | INFO | Thread-11853 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:25:01,101 | test | INFO | Thread-11853 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:24:42.370909421-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:24:42.331202677-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:22:05.397191305-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:22:05.327293778-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:25:01,102 | test | INFO | Thread-11853 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:25:01,104 | test | INFO | Thread-11853 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:24:42.454783928-07:00', 'error': "exit status 1: failed to lock archive; an old style lockfile still exists, if you are certain there isn't another cbbackupmgr process running remove '/tmp/my_archive/archive-plan9/lock.lk'", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:24:42.364546779-07:00', 'error': "exit status 1: failed to lock archive; an old style lockfile still exists, if you are certain there isn't another cbbackupmgr process running remove '/tmp/my_archive/archive-plan9/lock.lk'", 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:24:42.335874878-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:22:05.368246755-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:25:03,960 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 49.67 2021-04-14 07:25:09,003 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 57.14 2021-04-14 07:25:14,111 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 92.86 2021-04-14 07:25:24,170 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 07:25:24,203 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 191.775000095 sec 2021-04-14 07:25:46,203 | test | INFO | Thread-11855 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:25:46,204 | test | INFO | Thread-11855 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:25:29.076919534-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:25:29.040537335-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:23:11.785644167-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:22:05.664704066-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:25:48,105 | test | INFO | MainThread | [Collections:test_volume_taf:638] Collecting post_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root] 2021-04-14 07:26:07,141 | test | INFO | Thread-11856 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:26:39,232 | test | INFO | Thread-11857 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:27:07,938 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | bucket1 | couchbase | 2 | none | 0 | 2175 | 629145600 | 119853352 | 180717680 | | bucket2 | couchbase | 2 | none | 0 | 2175 | 1887436800 | 118910392 | 187475590 | | bucket3 | ephemeral | 2 | none | 0 | 2175 | 629145600 | 91409456 | 204 | | default | couchbase | 2 | none | 0 | 14232000 | 106954752000 | 11522834576 | 13882419746 | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ 2021-04-14 07:27:07,938 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 07:27:26,549 | test | INFO | MainThread | [Collections:test_volume_taf:567] Step 14: Graceful Failover a node and DeltaRecovery that node with data load in parallel 2021-04-14 07:27:26,638 | test | INFO | MainThread | [Collections:test_volume_taf:576] Collecting pre_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root] 2021-04-14 07:28:39,224 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 07:28:39,233 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 07:29:33,131 | test | INFO | Thread-12932 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:30:17,125 | test | INFO | Thread-12934 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:30:17,134 | test | INFO | Thread-12934 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:29:59.41189295-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:29:59.319920232-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'd54bc8af7e37630cd7d6f8e576460a3b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:26:26.491758775-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:22:05.456683712-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:30:39,029 | test | INFO | Thread-12935 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:30:39,033 | test | INFO | Thread-12935 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:30:28.911509976-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket2\': failed to transfer bucket data for bucket \'bucket2\': failed to transfer key value data: failed to transfer key value data: failed to initialise worker 2: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket2","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"13ad932c-8a0d-4701-c2e9-ca9ce7c1e39e","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.121.74:55218","last_connection_id":"76fba589c48e9345/d8bf12f2068f61fc"}', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:30:28.853393237-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket2\': failed to transfer bucket data for bucket \'bucket2\': failed to transfer key value data: failed to transfer key value data: failed to initialise worker 2: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket2","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"13ad932c-8a0d-4701-c2e9-ca9ce7c1e39e","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.121.74:55218","last_connection_id":"76fba589c48e9345/d8bf12f2068f61fc"}', 'error_code': 2, 'node_id': 'ecf2dc24125339004ba632ffc3d05508', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:29:06.2561539-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:22:05.793896458-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:31:27,525 | test | INFO | pool-3-thread-6 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******....................................................................................................*****X | 78.797 | | 42.602 | .....................................................******X | 42.602 | | 2.899 | ..******....*****X | 5.902 | | 3.000 | ..******.................******X | 16.101 | | 6.799 | .......******.......*****X | 12.101 | | 2.900 | ..******.******....*****X | 6.800 | | 0.000 | ************....******X | 3.401 | | 0.000 | ********X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:31:28,953 | test | INFO | pool-3-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******............*............................................................*****......******.................X | 85354.234 | | 83728.009 | .............................................................................................****X | 83728.009 | | 63243.302 | ......................................................................**X | 63243.302 | | 63243.302 | ......................................................................***.............................*****X | 89021.469 | | 84728.470 | ..............................................................................................*****X | 84728.470 | | 84175.690 | .............................................................................................*****X | 84175.690 | | 79058.782 | .......................................................................................*****X | 79058.782 | | 73360.815 | .................................................................................*****X | 73360.815 | | 70526.075 | ..............................................................................*****X | 70526.075 | | 68857.200 | ............................................................................****X | 68857.200 | | 68857.200 | ............................................................................X | 68857.200 | | 60003.800 | ..................................................................****X | 60003.800 | | 55144.245 | ............................................................X | 55144.245 | | 53974.745 | ...........................................................******..............******.................*****X | 81365.100 | | 79905.200 | ........................................................................................***X | 79905.200 | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ 2021-04-14 07:31:30,174 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******....................................................................................................*****X | 92.194 | | 18.202 | ..................*****X | 18.202 | | 12.197 | ............*****X | 12.197 | | 5.901 | .....******....*****X | 9.799 | | 0.000 | ******.............*****X | 12.198 | | 3.400 | ..*******.......**********X | 9.701 | | 0.000 | ************...******...******X | 6.301 | | 0.000 | ******...****X | 3.400 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:31:31,210 | test | INFO | pool-3-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******....................................................................................................*****X | 48.598 | | 17.901 | ...................................*****X | 17.901 | | 12.396 | ........................*****X | 12.396 | | 9.202 | .................******......******..*............****X | 18.701 | | 3.000 | .....*******............*****X | 8.901 | | 5.900 | ...........******..............****X | 12.801 | | 6.300 | ...........*****X | 6.300 | | 0.000 | ******..................******.*****X | 9.701 | | 3.400 | .....***********X | 3.400 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:31:42,664 | test | INFO | Thread-12938 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:31:42,665 | test | INFO | Thread-12938 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:31:27.146274712-07:00', 'error': 'exit status 1: ', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:31:27.086461218-07:00', 'error': 'exit status 1: ', 'error_code': 2, 'node_id': '30ff20027c87bcb7260bc9f7046b6daa', 'out': None, 'progress': 50.20566174691508, 'start': '2021-04-14T07:26:00.965553312-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:22:05.608696084-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:33:39,533 | test | INFO | MainThread | [Collections:wait_for_failover_or_assert:459] 1 nodes failed over as expected in 0.0549998283386 seconds 2021-04-14 07:33:39,877 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +----------------+------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+------------+-----------------------+----------------+--------------+ | 172.23.105.175 | backup, kv | 7.0.0-4907-enterprise | 9.85862156021 | Cluster node | | 172.23.106.236 | backup, kv | 7.0.0-4907-enterprise | 0.827378713802 | Cluster node | | 172.23.106.251 | backup, kv | 7.0.0-4907-enterprise | 5.50100704935 | Cluster node | | 172.23.106.233 | backup, kv | 7.0.0-4907-enterprise | 3.24487485851 | Cluster node | | 172.23.106.238 | backup, kv | 7.0.0-4907-enterprise | 3.13720549326 | Cluster node | | 172.23.121.74 | backup, kv | 7.0.0-4907-enterprise | 3.32200830502 | Cluster node | +----------------+------------+-----------------------+----------------+--------------+ 2021-04-14 07:33:39,885 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 07:33:44,924 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 07:33:49,966 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-14 07:33:55,009 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 3.6 2021-04-14 07:34:00,059 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 8.99 2021-04-14 07:34:05,723 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 11.81 2021-04-14 07:34:10,767 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 23.69 2021-04-14 07:34:15,822 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 29.72 2021-04-14 07:34:20,871 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 42.36 2021-04-14 07:34:25,917 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 53.97 2021-04-14 07:34:35,969 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-14 07:34:36,003 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 56.126999855 sec 2021-04-14 07:34:58,473 | test | INFO | MainThread | [Collections:test_volume_taf:638] Collecting post_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root] 2021-04-14 07:36:15,085 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ | bucket1 | couchbase | 2 | none | 0 | 2250 | 629145600 | 113909624 | 208992694 | | bucket2 | couchbase | 2 | none | 0 | 2250 | 1887436800 | 113953304 | 214367289 | | bucket3 | ephemeral | 2 | none | 0 | 2250 | 629145600 | 92295472 | 204 | | default | couchbase | 2 | none | 0 | 14801000 | 106954752000 | 10227319160 | 13225878137 | +---------+-----------+----------+------------+-----+----------+--------------+-------------+-------------+ 2021-04-14 07:36:15,086 | test | INFO | MainThread | [Collections:check_logs:115] Checking logs on [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.250 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root, ip:172.23.121.78 port:8091 ssh_username:root] 2021-04-14 07:36:31,003 | test | INFO | MainThread | [Collections:test_volume_taf:567] Step 15: Hard Failover a node and RebalanceOut that node with data load in parallel 2021-04-14 07:36:31,096 | test | INFO | MainThread | [Collections:test_volume_taf:576] Collecting pre_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.236 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root] 2021-04-14 07:37:38,384 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-14 07:37:38,391 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-14 07:37:58,336 | test | INFO | Thread-12956 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:37:58,339 | test | INFO | Thread-12956 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:37:41.855462841-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket2\': failed to transfer bucket data for bucket \'bucket2\': failed to transfer key value data: failed to transfer key value data: failed to initialise worker 0: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket2","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"b0278714-de7b-46f4-d867-0eb3f0baf0a5","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.106.251:58482","last_connection_id":"374dc2ac009351c4/73aae29d5589114c"}', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:37:41.819758958-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket2\': failed to transfer bucket data for bucket \'bucket2\': failed to transfer key value data: failed to transfer key value data: failed to initialise worker 0: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket2","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"b0278714-de7b-46f4-d867-0eb3f0baf0a5","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.106.251:58482","last_connection_id":"374dc2ac009351c4/73aae29d5589114c"}', 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:37:06.156333896-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:37:06.081884072-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:37:58,341 | test | INFO | Thread-12956 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:37:58,344 | test | INFO | Thread-12956 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:37:39.128673215-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket1\': failed to transfer bucket data for bucket \'bucket1\': failed to transfer key value data: failed to transfer key value data: failed to open stream: failed to stream vBucket 979: client received unexpected error \'EOF | {"bucket":"bucket1","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.106.238:52092","last_connection_id":"ebe68dd409b44f0a/9be3acbec5b6390f"}\'', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:37:39.056091056-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket1\': failed to transfer bucket data for bucket \'bucket1\': failed to transfer key value data: failed to transfer key value data: failed to open stream: failed to stream vBucket 979: client received unexpected error \'EOF | {"bucket":"bucket1","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.106.238:52092","last_connection_id":"ebe68dd409b44f0a/9be3acbec5b6390f"}\'', 'error_code': 2, 'node_id': '30ff20027c87bcb7260bc9f7046b6daa', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:37:05.935739676-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:37:05.849184833-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:37:58,345 | test | INFO | Thread-12956 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:37:58,346 | test | INFO | Thread-12956 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:37:41.398200985-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket2\': failed to transfer bucket data for bucket \'bucket2\': failed to transfer data range: failed to determine which data to transfer: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket2","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"e0cefcdb-a6e9-441b-b97e-5b7ae898cb0c","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.105.175:56908","last_connection_id":"02ba8a14af22b32f/dcd1031637d10d5c"}', 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:37:41.362472934-07:00', 'error': 'exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket \'bucket2\': failed to transfer bucket data for bucket \'bucket2\': failed to transfer data range: failed to determine which data to transfer: failed to get gocbcore DCP agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket2","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"e0cefcdb-a6e9-441b-b97e-5b7ae898cb0c","last_dispatched_to":"172.23.106.236:11210","last_dispatched_from":"172.23.105.175:56908","last_connection_id":"02ba8a14af22b32f/dcd1031637d10d5c"}', 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:37:06.002862887-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:37:05.886616738-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:37:58,348 | test | INFO | Thread-12956 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:37:58,351 | test | INFO | Thread-12956 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:37:39.718208494-07:00', 'error': "exit status 1: failed to lock archive; an old style lockfile still exists, if you are certain there isn't another cbbackupmgr process running remove '/tmp/my_archive/archive-plan9/lock.lk'", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:37:39.091951817-07:00', 'error': "exit status 1: failed to lock archive; an old style lockfile still exists, if you are certain there isn't another cbbackupmgr process running remove '/tmp/my_archive/archive-plan9/lock.lk'", 'error_code': 2, 'node_id': '30ff20027c87bcb7260bc9f7046b6daa', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:37:39.060272649-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:37:06.025522795-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:40:04,200 | test | INFO | pool-3-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******....................................................................................................*****X | 37.598 | | 34.598 | ...........................................................................................X | 34.598 | | 6.600 | ................******........................******................*****X | 21.600 | | 9.700 | ........................*****X | 9.700 | | 6.900 | .................X | 6.900 | | 6.200 | ...............*****X | 6.200 | | 6.100 | ...............******..........................................................******X | 28.000 | | 12.700 | ................................******...........******X | 16.800 | | 7.200 | ..................******......*****X | 9.599 | | 6.501 | ................*..*****X | 7.201 | | 7.199 | ..................X | 7.199 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:40:04,967 | test | INFO | pool-3-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +------------+-----------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +------------+-----------------------------------------------------------------------------------------------------------------------+------------+ | 0.000 | ******............................................................******.......................................*****X | 113324.388 | | 105221.700 | ...........................................................................................*****X | 105221.700 | | 97689.100 | .....................................................................................******X | 97689.100 | | 97214.450 | ....................................................................................*****X | 97214.450 | | 91306.732 | ...............................................................................**X | 91306.732 | | 91306.732 | ...............................................................................**X | 91306.732 | | 91306.732 | ...............................................................................X | 91306.732 | | 90827.100 | ...............................................................................*****X | 90827.100 | | 87736.000 | ............................................................................*****X | 87736.000 | | 84393.300 | .........................................................................****X | 84393.300 | | 84393.300 | .........................................................................*X | 84393.300 | | 82309.600 | .......................................................................*****X | 82309.600 | | 82270.100 | .......................................................................X | 82270.100 | | 82035.600 | .......................................................................**X | 82035.600 | | 82035.600 | .......................................................................*X | 82035.600 | | 81435.168 | ......................................................................X | 81435.168 | | 81304.568 | ......................................................................*X | 81304.568 | | 81304.568 | ......................................................................**X | 81304.568 | | 81304.568 | ......................................................................******X | 81418.785 | | 79895.908 | .....................................................................X | 79895.908 | | 79539.308 | .....................................................................X | 79539.308 | +------------+-----------------------------------------------------------------------------------------------------------------------+------------+ 2021-04-14 07:40:06,240 | test | INFO | pool-3-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****....................................................................................*................*****X | 75.496 | | 6.200 | .......******.................*****X | 19.000 | | 12.200 | ...............******.*****X | 12.900 | | 10.800 | .............X | 10.800 | | 10.200 | ............*****X | 10.200 | | 9.600 | ...........******X | 9.800 | | 9.700 | ...........****X | 9.700 | | 9.100 | ...........******X | 9.100 | | 3.600 | ...*****X | 3.600 | | 0.000 | *******..............*******...*****X | 13.202 | | 0.000 | *X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:40:07,214 | test | INFO | pool-3-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.........................................................................................*..........***.*X | 84.296 | | 84.296 | ..................................................................................................X | 84.296 | | 3.000 | ..******...........******..*.........****X | 21.900 | | 7.200 | .......*******..........*****X | 15.900 | | 6.600 | ......******.............******X | 16.900 | | 3.000 | ..******.....*****X | 7.200 | | 1.800 | .X | 1.800 | | 0.000 | ******............*****X | 10.199 | | 0.000 | *******....**X | 3.600 | +--------+-----------------------------------------------------------------------------------------------------------------+--------+ 2021-04-14 07:40:28,382 | test | INFO | Thread-14030 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:40:28,387 | test | INFO | Thread-14030 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:40:21.099847265-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:40:20.988460784-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': 'a124c621aa091a6dc0c789af5a5b1e63', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:37:41.822627439-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:37:06.12077248-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:42:20,994 | test | INFO | Thread-14035 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:42:20,997 | test | INFO | Thread-14035 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:42:00.550983864-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:42:00.518911854-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to close vBucket: failed to commit final chunk data to disk: failed to commit index: failed to commit index transaction: disk I/O error", 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:37:41.367481377-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:37:06.24134673-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:42:41,417 | test | INFO | MainThread | [Collections:wait_for_failover_or_assert:459] 1 nodes failed over as expected in 0.0510001182556 seconds 2021-04-14 07:45:02,075 | test | INFO | Thread-14042 | [backup_service:monitor:87] The backup service task task0 has completed with status: failed. (Disallowed) 2021-04-14 07:45:02,078 | test | INFO | Thread-14042 | [backup_service:monitor:92] Logged disallowed task: {'end': '2021-04-14T07:44:45.890432886-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to retry DCP stream: failed to stream vBucket 818: client received rollback, either purge this backup or create a new backup repository. See the ROLLBACK section in the backup documentation for more information", 'error_code': 2, 'node_runs': [{'end': '2021-04-14T07:44:45.844709022-07:00', 'error': "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to retry DCP stream: failed to stream vBucket 818: client received rollback, either purge this backup or create a new backup repository. See the ROLLBACK section in the backup documentation for more information", 'error_code': 2, 'node_id': '30ea76886df1410255f3aea674a7b29b', 'out': None, 'progress': 0.0, 'start': '2021-04-14T07:42:00.522230795-07:00', 'status': 'failed'}], 'start': '2021-04-14T07:37:06.284566251-07:00', 'status': 'failed', 'target_cluster': None, 'task_name': 'task0', 'type': 'BACKUP'} 2021-04-14 07:45:02,078 | test | INFO | Thread-14042 | [backup_service:monitor:87] The backup service task task0 has completed with status: done. (Allowed) 2021-04-14 07:45:17,924 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 156.384000063 seconds to complete 2021-04-14 07:45:17,930 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-04-14 07:45:43,551 | test | INFO | MainThread | [Collections:test_volume_taf:638] Collecting post_failover_stats. KV nodes are [ip:172.23.105.175 port:8091 ssh_username:root, ip:172.23.106.233 port:8091 ssh_username:root, ip:172.23.106.238 port:8091 ssh_username:root, ip:172.23.106.251 port:8091 ssh_username:root, ip:172.23.121.74 port:8091 ssh_username:root] Build was aborted Aborted by Asad Zaidi Archiving artifacts Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: ABORTED