Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s83002 (P0_win_upgrade jython_slave) in workspace /data/workspace/temp_centos7_4node_jython Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/temp_centos7_4node_jython # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/neo^{commit} # timeout=10 Checking out Revision e1d814077b808021219e6c99beada4f93d4d9a8a (origin/neo) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f e1d814077b808021219e6c99beada4f93d4d9a8a > /usr/bin/git rev-list 92511000f36aff07377ea2dc5e93922da78f2b9f # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #3021 jenkins-temp_centos7_4node_jython-3021 # timeout=10 Run condition [Always] enabling prebuild for step [BuilderChain] No emails were triggered. [temp_centos7_4node_jython] $ /bin/bash -x /tmp/jenkins7030999451631790930.sh + jython_bin=/opt/jython/bin + jython_path=/opt/jython/bin/jython + ini_file=node.ini + git clone https://github.com/couchbaselabs/guides.git Cloning into 'guides'... + echo ' [global] username:root password:couchbase port:8091 # data_path:/data [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 [_1] ip:172.23.105.187 [_2] ip:172.23.123.22 [_3] ip:172.23.108.25 [_4] ip:172.23.105.98 ' + echo 'bucket_collections.rollback_tests.RollbackTests: test_rollback_n_times,nodes_init=4,replicas=3,num_items=10000,doc_ops=create,bucket_spec=single_bucket.three_scopes_two_collections,override_spec_params=replicas;num_items,infra_log_level=debug,log_level=debug ' + rm -rf 'logs/*' jars + sdk_client_params='-P transaction_version=1.2.1 -P client_version=3.4.7' + git submodule init + git submodule update --init --force --remote + '[' false == true ']' + rm -f /tmp/test.pcap + '[' false == true ']' + '[' '-p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True' == '' ']' + test_params='-p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919' + '[' TAF == testrunner ']' + '[' TAF == TAF ']' + '[' '' '!=' '' ']' + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.2.1 -P client_version=3.4.7 -P 'args=-i node.ini -c debug.conf -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919 -m rest' + tee logs/test_run.log tee: logs/test_run.log: No such file or directory Starting a Gradle Daemon, 5 busy Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.2.1 Using Java_client :: 3.4.7 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.20.3/38d33006f9c64f4c3703f80297c09b21cb0a7c6a/azure-storage-blob-12.20.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.4.7/498bc21fbb38a59cb49c2823dfc8e2727a41ebfa/java-client-3.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.5.3/84797e7319564a9f9ae7f4acbb12a3dc4c26d606/azure-storage-internal-avro-12.5.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.19.3/3e30b1be613815b10ea8d2cfa39f560aaeef5f0d/azure-storage-common-12.19.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.13.0/57474ba0daa16690465c32f1376fb4b274e00778/azure-core-http-netty-1.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.36.0/a1bd31cb23b2c6bc18c3fb2e9635c766ecdbe76/azure-core-1.36.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.13.4/e6d820112871f33cd94a1dcc54eef58874753b5/jackson-datatype-jsr310-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.4.2/325c06bdfeb628cfb80ebaaf1a26cc1eb558a585/jackson-databind-2.13.4.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.13.4/858c6cc78e1f08a885b1613e1d817c829df70a6e/jackson-annotations-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.13.4/cf934c681294b97ef6d80082faeefbe1edadf56/jackson-core-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.4/b739978806ffc80967ba0efe43b1296c29c4cfe8/jackson-dataformat-xml-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.4.0/c47579857bbf12c85499f431d4ecf27d77976b7c/woodstox-core-6.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.4.7/969a9c5ad4187b51945792919b1d2a21ec4b3054/core-io-2.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.27/83a4a8ba02ecbdb681dcc083479ff251f620d2a7/reactor-netty-http-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.27/afccb76721a9f2a982fb2b5f043aa2f70ecb00aa/reactor-netty-core-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.5.0/83749d14f5795905e5b8e8c258c917d4707b586c/reactor-core-3.5.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.87.Final/8079f16d6209e10f5d3a09522e7aa9ffb217e17e/netty-handler-proxy-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.87.Final/72b756ff290d782c9ebb36db7d42ee272270c0b4/netty-codec-http2-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.87.Final/3230e95d6832ee4306fe1ca774c7bcecf1da4b28/netty-codec-http-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.86.Final/cbec1f5c35a1466a8d998f3419c5c493ef5063ee/netty-resolver-dns-native-macos-4.1.86.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-classes-macos/4.1.86.Final/5d72f1896cb9dab1fb7c41470cb30a9e63a65643/netty-resolver-dns-classes-macos-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.86.Final/a1e2ef79e4944b5d38092328c36c68e677a4b5f3/netty-resolver-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.87.Final/2bd97491c22ebea4670c00f1bd5dbf65a8a1cfe7/netty-handler-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.87.Final/109027470b72d56914d76154e947f9bd5844bb9b/netty-codec-socks-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.86.Final/c8de479f36a8457541fcbb0016c851bde3e67693/netty-codec-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.87.Final/d7ab7363f736114e6324ccf802d094529b30b8d8/netty-codec-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.87.Final/3374b1ee72cfff2858821de824ca02d08aedee7c/netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.87.Final/28db44b32b4c68f14858e9d691646a7262b57863/netty-transport-native-kqueue-4.1.87.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-epoll/4.1.87.Final/86e61c5743940d80e10609746784ffa0e4642065/netty-transport-classes-epoll-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-kqueue/4.1.87.Final/5476926df0ad650d3287d61b0bd17f145ff13791/netty-transport-classes-kqueue-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.87.Final/4bf8308dc7d6ba2570681eaeddb16f8d1196b438/netty-transport-native-unix-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.87.Final/30c78c8ced3417f35e2a55f7533dc2bb43fef2aa/netty-transport-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.87.Final/8942174141fb0586e7fffefc8ae48e1458293696/netty-buffer-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.56.Final/6aa4a3f9e415daa973ef3d7efeb8807b15631faf/netty-tcnative-boringssl-static-2.0.56.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.4/3864a1320d97d7b045f729a326e1e077661f31b7/reactive-streams-1.0.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.87.Final/eaa964e16a67914c8d9b814d29a4b969635d72a0/netty-resolver-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.87.Final/3e7e80dd6e604144781fcb859b79cfe8d3730079/netty-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-classes/2.0.56.Final/df828615a3a4b4b44d7f41aa3d22b744fe7170d/netty-tcnative-classes-2.0.56.Final.jar scripts/ssh.py -i node.ini -c debug.conf -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.20.3/38d33006f9c64f4c3703f80297c09b21cb0a7c6a/azure-storage-blob-12.20.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.4.7/498bc21fbb38a59cb49c2823dfc8e2727a41ebfa/java-client-3.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.5.3/84797e7319564a9f9ae7f4acbb12a3dc4c26d606/azure-storage-internal-avro-12.5.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.19.3/3e30b1be613815b10ea8d2cfa39f560aaeef5f0d/azure-storage-common-12.19.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.13.0/57474ba0daa16690465c32f1376fb4b274e00778/azure-core-http-netty-1.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.36.0/a1bd31cb23b2c6bc18c3fb2e9635c766ecdbe76/azure-core-1.36.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.13.4/e6d820112871f33cd94a1dcc54eef58874753b5/jackson-datatype-jsr310-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.4.2/325c06bdfeb628cfb80ebaaf1a26cc1eb558a585/jackson-databind-2.13.4.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.13.4/858c6cc78e1f08a885b1613e1d817c829df70a6e/jackson-annotations-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.13.4/cf934c681294b97ef6d80082faeefbe1edadf56/jackson-core-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.4/b739978806ffc80967ba0efe43b1296c29c4cfe8/jackson-dataformat-xml-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.4.0/c47579857bbf12c85499f431d4ecf27d77976b7c/woodstox-core-6.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.4.7/969a9c5ad4187b51945792919b1d2a21ec4b3054/core-io-2.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.27/83a4a8ba02ecbdb681dcc083479ff251f620d2a7/reactor-netty-http-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.27/afccb76721a9f2a982fb2b5f043aa2f70ecb00aa/reactor-netty-core-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.5.0/83749d14f5795905e5b8e8c258c917d4707b586c/reactor-core-3.5.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.87.Final/8079f16d6209e10f5d3a09522e7aa9ffb217e17e/netty-handler-proxy-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.87.Final/72b756ff290d782c9ebb36db7d42ee272270c0b4/netty-codec-http2-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.87.Final/3230e95d6832ee4306fe1ca774c7bcecf1da4b28/netty-codec-http-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.86.Final/cbec1f5c35a1466a8d998f3419c5c493ef5063ee/netty-resolver-dns-native-macos-4.1.86.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-classes-macos/4.1.86.Final/5d72f1896cb9dab1fb7c41470cb30a9e63a65643/netty-resolver-dns-classes-macos-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.86.Final/a1e2ef79e4944b5d38092328c36c68e677a4b5f3/netty-resolver-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.87.Final/2bd97491c22ebea4670c00f1bd5dbf65a8a1cfe7/netty-handler-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.87.Final/109027470b72d56914d76154e947f9bd5844bb9b/netty-codec-socks-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.86.Final/c8de479f36a8457541fcbb0016c851bde3e67693/netty-codec-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.87.Final/d7ab7363f736114e6324ccf802d094529b30b8d8/netty-codec-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.87.Final/3374b1ee72cfff2858821de824ca02d08aedee7c/netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.87.Final/28db44b32b4c68f14858e9d691646a7262b57863/netty-transport-native-kqueue-4.1.87.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-epoll/4.1.87.Final/86e61c5743940d80e10609746784ffa0e4642065/netty-transport-classes-epoll-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-kqueue/4.1.87.Final/5476926df0ad650d3287d61b0bd17f145ff13791/netty-transport-classes-kqueue-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.87.Final/4bf8308dc7d6ba2570681eaeddb16f8d1196b438/netty-transport-native-unix-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.87.Final/30c78c8ced3417f35e2a55f7533dc2bb43fef2aa/netty-transport-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.87.Final/8942174141fb0586e7fffefc8ae48e1458293696/netty-buffer-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.56.Final/6aa4a3f9e415daa973ef3d7efeb8807b15631faf/netty-tcnative-boringssl-static-2.0.56.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.4/3864a1320d97d7b045f729a326e1e077661f31b7/reactive-streams-1.0.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.87.Final/eaa964e16a67914c8d9b814d29a4b969635d72a0/netty-resolver-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.87.Final/3e7e80dd6e604144781fcb859b79cfe8d3730079/netty-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-classes/2.0.56.Final/df828615a3a4b4b44d7f41aa3d22b744fe7170d/netty-tcnative-classes-2.0.56.Final.jar scripts/eagles_all_around.py -i node.ini -c debug.conf -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.20.3/38d33006f9c64f4c3703f80297c09b21cb0a7c6a/azure-storage-blob-12.20.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.4.7/498bc21fbb38a59cb49c2823dfc8e2727a41ebfa/java-client-3.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.5.3/84797e7319564a9f9ae7f4acbb12a3dc4c26d606/azure-storage-internal-avro-12.5.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.19.3/3e30b1be613815b10ea8d2cfa39f560aaeef5f0d/azure-storage-common-12.19.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.13.0/57474ba0daa16690465c32f1376fb4b274e00778/azure-core-http-netty-1.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.36.0/a1bd31cb23b2c6bc18c3fb2e9635c766ecdbe76/azure-core-1.36.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.13.4/e6d820112871f33cd94a1dcc54eef58874753b5/jackson-datatype-jsr310-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.4.2/325c06bdfeb628cfb80ebaaf1a26cc1eb558a585/jackson-databind-2.13.4.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.13.4/858c6cc78e1f08a885b1613e1d817c829df70a6e/jackson-annotations-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.13.4/cf934c681294b97ef6d80082faeefbe1edadf56/jackson-core-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.4/b739978806ffc80967ba0efe43b1296c29c4cfe8/jackson-dataformat-xml-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.4.0/c47579857bbf12c85499f431d4ecf27d77976b7c/woodstox-core-6.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.4.7/969a9c5ad4187b51945792919b1d2a21ec4b3054/core-io-2.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.27/83a4a8ba02ecbdb681dcc083479ff251f620d2a7/reactor-netty-http-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.27/afccb76721a9f2a982fb2b5f043aa2f70ecb00aa/reactor-netty-core-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.5.0/83749d14f5795905e5b8e8c258c917d4707b586c/reactor-core-3.5.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.87.Final/8079f16d6209e10f5d3a09522e7aa9ffb217e17e/netty-handler-proxy-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.87.Final/72b756ff290d782c9ebb36db7d42ee272270c0b4/netty-codec-http2-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.87.Final/3230e95d6832ee4306fe1ca774c7bcecf1da4b28/netty-codec-http-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.86.Final/cbec1f5c35a1466a8d998f3419c5c493ef5063ee/netty-resolver-dns-native-macos-4.1.86.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-classes-macos/4.1.86.Final/5d72f1896cb9dab1fb7c41470cb30a9e63a65643/netty-resolver-dns-classes-macos-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.86.Final/a1e2ef79e4944b5d38092328c36c68e677a4b5f3/netty-resolver-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.87.Final/2bd97491c22ebea4670c00f1bd5dbf65a8a1cfe7/netty-handler-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.87.Final/109027470b72d56914d76154e947f9bd5844bb9b/netty-codec-socks-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.86.Final/c8de479f36a8457541fcbb0016c851bde3e67693/netty-codec-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.87.Final/d7ab7363f736114e6324ccf802d094529b30b8d8/netty-codec-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.87.Final/3374b1ee72cfff2858821de824ca02d08aedee7c/netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.87.Final/28db44b32b4c68f14858e9d691646a7262b57863/netty-transport-native-kqueue-4.1.87.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-epoll/4.1.87.Final/86e61c5743940d80e10609746784ffa0e4642065/netty-transport-classes-epoll-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-kqueue/4.1.87.Final/5476926df0ad650d3287d61b0bd17f145ff13791/netty-transport-classes-kqueue-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.87.Final/4bf8308dc7d6ba2570681eaeddb16f8d1196b438/netty-transport-native-unix-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.87.Final/30c78c8ced3417f35e2a55f7533dc2bb43fef2aa/netty-transport-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.87.Final/8942174141fb0586e7fffefc8ae48e1458293696/netty-buffer-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.56.Final/6aa4a3f9e415daa973ef3d7efeb8807b15631faf/netty-tcnative-boringssl-static-2.0.56.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.4/3864a1320d97d7b045f729a326e1e077661f31b7/reactive-streams-1.0.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.87.Final/eaa964e16a67914c8d9b814d29a4b969635d72a0/netty-resolver-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.87.Final/3e7e80dd6e604144781fcb859b79cfe8d3730079/netty-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-classes/2.0.56.Final/df828615a3a4b4b44d7f41aa3d22b744fe7170d/netty-tcnative-classes-2.0.56.Final.jar:build/classes/java/main scripts/install.py -i node.ini -c debug.conf -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.20.3/38d33006f9c64f4c3703f80297c09b21cb0a7c6a/azure-storage-blob-12.20.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.4.7/498bc21fbb38a59cb49c2823dfc8e2727a41ebfa/java-client-3.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.5.3/84797e7319564a9f9ae7f4acbb12a3dc4c26d606/azure-storage-internal-avro-12.5.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.19.3/3e30b1be613815b10ea8d2cfa39f560aaeef5f0d/azure-storage-common-12.19.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.13.0/57474ba0daa16690465c32f1376fb4b274e00778/azure-core-http-netty-1.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.36.0/a1bd31cb23b2c6bc18c3fb2e9635c766ecdbe76/azure-core-1.36.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.13.4/e6d820112871f33cd94a1dcc54eef58874753b5/jackson-datatype-jsr310-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.4.2/325c06bdfeb628cfb80ebaaf1a26cc1eb558a585/jackson-databind-2.13.4.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.13.4/858c6cc78e1f08a885b1613e1d817c829df70a6e/jackson-annotations-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.13.4/cf934c681294b97ef6d80082faeefbe1edadf56/jackson-core-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.4/b739978806ffc80967ba0efe43b1296c29c4cfe8/jackson-dataformat-xml-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.4.0/c47579857bbf12c85499f431d4ecf27d77976b7c/woodstox-core-6.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.4.7/969a9c5ad4187b51945792919b1d2a21ec4b3054/core-io-2.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.27/83a4a8ba02ecbdb681dcc083479ff251f620d2a7/reactor-netty-http-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.27/afccb76721a9f2a982fb2b5f043aa2f70ecb00aa/reactor-netty-core-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.5.0/83749d14f5795905e5b8e8c258c917d4707b586c/reactor-core-3.5.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.87.Final/8079f16d6209e10f5d3a09522e7aa9ffb217e17e/netty-handler-proxy-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.87.Final/72b756ff290d782c9ebb36db7d42ee272270c0b4/netty-codec-http2-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.87.Final/3230e95d6832ee4306fe1ca774c7bcecf1da4b28/netty-codec-http-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.86.Final/cbec1f5c35a1466a8d998f3419c5c493ef5063ee/netty-resolver-dns-native-macos-4.1.86.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-classes-macos/4.1.86.Final/5d72f1896cb9dab1fb7c41470cb30a9e63a65643/netty-resolver-dns-classes-macos-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.86.Final/a1e2ef79e4944b5d38092328c36c68e677a4b5f3/netty-resolver-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.87.Final/2bd97491c22ebea4670c00f1bd5dbf65a8a1cfe7/netty-handler-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.87.Final/109027470b72d56914d76154e947f9bd5844bb9b/netty-codec-socks-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.86.Final/c8de479f36a8457541fcbb0016c851bde3e67693/netty-codec-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.87.Final/d7ab7363f736114e6324ccf802d094529b30b8d8/netty-codec-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.87.Final/3374b1ee72cfff2858821de824ca02d08aedee7c/netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.87.Final/28db44b32b4c68f14858e9d691646a7262b57863/netty-transport-native-kqueue-4.1.87.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-epoll/4.1.87.Final/86e61c5743940d80e10609746784ffa0e4642065/netty-transport-classes-epoll-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-kqueue/4.1.87.Final/5476926df0ad650d3287d61b0bd17f145ff13791/netty-transport-classes-kqueue-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.87.Final/4bf8308dc7d6ba2570681eaeddb16f8d1196b438/netty-transport-native-unix-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.87.Final/30c78c8ced3417f35e2a55f7533dc2bb43fef2aa/netty-transport-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.87.Final/8942174141fb0586e7fffefc8ae48e1458293696/netty-buffer-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.56.Final/6aa4a3f9e415daa973ef3d7efeb8807b15631faf/netty-tcnative-boringssl-static-2.0.56.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.4/3864a1320d97d7b045f729a326e1e077661f31b7/reactive-streams-1.0.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.87.Final/eaa964e16a67914c8d9b814d29a4b969635d72a0/netty-resolver-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.87.Final/3e7e80dd6e604144781fcb859b79cfe8d3730079/netty-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-classes/2.0.56.Final/df828615a3a4b4b44d7f41aa3d22b744fe7170d/netty-tcnative-classes-2.0.56.Final.jar:build/classes/java/main:src/main/resources testrunner.py -i node.ini -c debug.conf -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.20.3/38d33006f9c64f4c3703f80297c09b21cb0a7c6a/azure-storage-blob-12.20.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.4.7/498bc21fbb38a59cb49c2823dfc8e2727a41ebfa/java-client-3.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.5.3/84797e7319564a9f9ae7f4acbb12a3dc4c26d606/azure-storage-internal-avro-12.5.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.19.3/3e30b1be613815b10ea8d2cfa39f560aaeef5f0d/azure-storage-common-12.19.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.13.0/57474ba0daa16690465c32f1376fb4b274e00778/azure-core-http-netty-1.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.36.0/a1bd31cb23b2c6bc18c3fb2e9635c766ecdbe76/azure-core-1.36.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.13.4/e6d820112871f33cd94a1dcc54eef58874753b5/jackson-datatype-jsr310-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.4.2/325c06bdfeb628cfb80ebaaf1a26cc1eb558a585/jackson-databind-2.13.4.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.13.4/858c6cc78e1f08a885b1613e1d817c829df70a6e/jackson-annotations-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.13.4/cf934c681294b97ef6d80082faeefbe1edadf56/jackson-core-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.13.4/b739978806ffc80967ba0efe43b1296c29c4cfe8/jackson-dataformat-xml-2.13.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.4.0/c47579857bbf12c85499f431d4ecf27d77976b7c/woodstox-core-6.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.4.7/969a9c5ad4187b51945792919b1d2a21ec4b3054/core-io-2.4.7.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.27/83a4a8ba02ecbdb681dcc083479ff251f620d2a7/reactor-netty-http-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.27/afccb76721a9f2a982fb2b5f043aa2f70ecb00aa/reactor-netty-core-1.0.27.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.5.0/83749d14f5795905e5b8e8c258c917d4707b586c/reactor-core-3.5.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.87.Final/8079f16d6209e10f5d3a09522e7aa9ffb217e17e/netty-handler-proxy-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.87.Final/72b756ff290d782c9ebb36db7d42ee272270c0b4/netty-codec-http2-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.87.Final/3230e95d6832ee4306fe1ca774c7bcecf1da4b28/netty-codec-http-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.86.Final/cbec1f5c35a1466a8d998f3419c5c493ef5063ee/netty-resolver-dns-native-macos-4.1.86.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-classes-macos/4.1.86.Final/5d72f1896cb9dab1fb7c41470cb30a9e63a65643/netty-resolver-dns-classes-macos-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.86.Final/a1e2ef79e4944b5d38092328c36c68e677a4b5f3/netty-resolver-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.87.Final/2bd97491c22ebea4670c00f1bd5dbf65a8a1cfe7/netty-handler-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.87.Final/109027470b72d56914d76154e947f9bd5844bb9b/netty-codec-socks-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.86.Final/c8de479f36a8457541fcbb0016c851bde3e67693/netty-codec-dns-4.1.86.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.87.Final/d7ab7363f736114e6324ccf802d094529b30b8d8/netty-codec-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.87.Final/3374b1ee72cfff2858821de824ca02d08aedee7c/netty-transport-native-epoll-4.1.87.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.87.Final/28db44b32b4c68f14858e9d691646a7262b57863/netty-transport-native-kqueue-4.1.87.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-epoll/4.1.87.Final/86e61c5743940d80e10609746784ffa0e4642065/netty-transport-classes-epoll-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-kqueue/4.1.87.Final/5476926df0ad650d3287d61b0bd17f145ff13791/netty-transport-classes-kqueue-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.87.Final/4bf8308dc7d6ba2570681eaeddb16f8d1196b438/netty-transport-native-unix-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.87.Final/30c78c8ced3417f35e2a55f7533dc2bb43fef2aa/netty-transport-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.87.Final/8942174141fb0586e7fffefc8ae48e1458293696/netty-buffer-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.56.Final/6aa4a3f9e415daa973ef3d7efeb8807b15631faf/netty-tcnative-boringssl-static-2.0.56.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.4/3864a1320d97d7b045f729a326e1e077661f31b7/reactive-streams-1.0.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.87.Final/eaa964e16a67914c8d9b814d29a4b969635d72a0/netty-resolver-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.87.Final/3e7e80dd6e604144781fcb859b79cfe8d3730079/netty-common-4.1.87.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-classes/2.0.56.Final/df828615a3a4b4b44d7f41aa3d22b744fe7170d/netty-tcnative-classes-2.0.56.Final.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i node.ini -c debug.conf -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919 -m rest > Task :compileJavawarning: unknown enum constant When.MAYBE reason: class file for javax.annotation.meta.When not found Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 1 warning > Task :testrunner Filename: debug.conf Prefix: bucket_collections.rollback_tests.RollbackTests Global Test input params: {'cluster_name': 'node', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'ini': 'node.ini', 'num_nodes': 4, 'rerun': 'False', 'skip_cluster_reset': 'True', 'skip_collections_cleanup': 'True', 'spec': 'debug', 'upgrade_version': '7.2.1-5919'} Logs will be stored at /data/workspace/temp_centos7_4node_jython/logs/testrunner-23-Aug-22_22-04-19/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,skip_collections_cleanup=True,upgrade_version=7.2.1-5919 -t bucket_collections.rollback_tests.RollbackTests.test_rollback_n_times,nodes_init=4,replicas=3,num_items=10000,doc_ops=create,bucket_spec=single_bucket.three_scopes_two_collections,override_spec_params=replicas;num_items,infra_log_level=debug,log_level=debug' Test Input params: {'cluster_name': 'node', 'ini': 'node.ini', 'get-cbcollect-info': 'False', 'replicas': '3', 'log_level': 'debug', 'conf_file': 'debug.conf', 'upgrade_version': '7.2.1-5919', 'spec': 'debug', 'rerun': 'False', 'num_nodes': 4, 'logs_folder': '/data/workspace/temp_centos7_4node_jython/logs/testrunner-23-Aug-22_22-04-19/test_1', 'nodes_init': '4', 'skip_cluster_reset': 'True', 'doc_ops': 'create', 'num_items': '10000', 'skip_collections_cleanup': 'True', 'bucket_spec': 'single_bucket.three_scopes_two_collections', 'case_number': 1, 'override_spec_params': 'replicas;num_items', 'infra_log_level': 'debug'} test_rollback_n_times (bucket_collections.rollback_tests.RollbackTests) ... 2023-08-22 22:04:24,250 | infra | INFO | pool-5-thread-5 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:[ip:172.23.105.187 port:8091 ssh_username:root] Kwds:{}' started 2023-08-22 22:04:24,250 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:[ip:172.23.105.187 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:24,259 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:[ip:172.23.123.22 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:24,259 | infra | INFO | pool-5-thread-6 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:[ip:172.23.123.22 port:8091 ssh_username:root] Kwds:{}' started 2023-08-22 22:04:24,260 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:[ip:172.23.108.25 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:24,262 | infra | INFO | pool-5-thread-7 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:[ip:172.23.108.25 port:8091 ssh_username:root] Kwds:{}' started 2023-08-22 22:04:24,263 | infra | INFO | pool-5-thread-8 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:[ip:172.23.105.98 port:8091 ssh_username:root] Kwds:{}' started 2023-08-22 22:04:24,263 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:[ip:172.23.105.98 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:24,266 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:[ip:172.23.105.187 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:24,273 | infra | DEBUG | pool-5-thread-5 | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:04:24,273 | infra | DEBUG | pool-5-thread-6 | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:04:24,276 | infra | DEBUG | pool-5-thread-8 | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:04:24,276 | infra | DEBUG | pool-5-thread-7 | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:04:24,351 | infra | DEBUG | pool-5-thread-6 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover 0 2023-08-22 22:04:24,365 | infra | DEBUG | pool-5-thread-5 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover 0 2023-08-22 22:04:24,375 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover 0 2023-08-22 22:04:24,377 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost:8091 -u Administrator -p password --enable-auto-failover 0 2023-08-22 22:04:24,713 | infra | INFO | pool-5-thread-6 | [node_ready_functions:_disable_tls:186] 172.23.123.22: ['SUCCESS: Auto-failover settings modified\n'] 2023-08-22 22:04:24,714 | infra | DEBUG | pool-5-thread-6 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/couchbase-cli setting-security -c localhost:8091 -u Administrator -p password --set --cluster-encryption-level control 2023-08-22 22:04:24,855 | infra | INFO | pool-5-thread-8 | [node_ready_functions:_disable_tls:186] 172.23.105.98: ['SUCCESS: Auto-failover settings modified\n'] 2023-08-22 22:04:24,857 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/couchbase-cli setting-security -c localhost:8091 -u Administrator -p password --set --cluster-encryption-level control 2023-08-22 22:04:24,858 | infra | INFO | pool-5-thread-5 | [node_ready_functions:_disable_tls:186] 172.23.105.187: ['SUCCESS: Auto-failover settings modified\n'] 2023-08-22 22:04:24,859 | infra | DEBUG | pool-5-thread-5 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/couchbase-cli setting-security -c localhost:8091 -u Administrator -p password --set --cluster-encryption-level control 2023-08-22 22:04:24,872 | infra | INFO | pool-5-thread-7 | [node_ready_functions:_disable_tls:186] 172.23.108.25: ['SUCCESS: Auto-failover settings modified\n'] 2023-08-22 22:04:24,874 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/couchbase-cli setting-security -c localhost:8091 -u Administrator -p password --set --cluster-encryption-level control 2023-08-22 22:04:25,006 | infra | INFO | pool-5-thread-6 | [node_ready_functions:_disable_tls:188] 172.23.123.22: ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled.\n"] 2023-08-22 22:04:25,006 | infra | DEBUG | pool-5-thread-6 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost:8091 -u Administrator -p password --disable 2023-08-22 22:04:25,259 | infra | INFO | pool-5-thread-8 | [node_ready_functions:_disable_tls:188] 172.23.105.98: ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled.\n"] 2023-08-22 22:04:25,259 | infra | INFO | pool-5-thread-7 | [node_ready_functions:_disable_tls:188] 172.23.108.25: ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled.\n"] 2023-08-22 22:04:25,259 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost:8091 -u Administrator -p password --disable 2023-08-22 22:04:25,260 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost:8091 -u Administrator -p password --disable 2023-08-22 22:04:25,270 | infra | INFO | pool-5-thread-5 | [node_ready_functions:_disable_tls:188] 172.23.105.187: ["ERROR: clusterEncryptionLevel - Can't set cluster encryption level when cluster encryption is disabled.\n"] 2023-08-22 22:04:25,272 | infra | DEBUG | pool-5-thread-5 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost:8091 -u Administrator -p password --disable 2023-08-22 22:04:25,470 | infra | INFO | pool-5-thread-6 | [node_ready_functions:_disable_tls:190] 172.23.123.22: ['Turned off encryption for node: http://172.23.105.187:8091\n', 'Turned off encryption for node: http://172.23.105.98:8091\n', 'Turned off encryption for node: http://172.23.108.25:8091\n', 'Turned off encryption for node: http://172.23.123.22:8091\n', 'SUCCESS: Switched node-to-node encryption off\n'] 2023-08-22 22:04:25,470 | infra | DEBUG | pool-5-thread-6 | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.123.22 2023-08-22 22:04:25,471 | infra | INFO | pool-5-thread-6 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:[ip:172.23.123.22 port:8091 ssh_username:root] Kwds:{}' completed 2023-08-22 22:04:25,834 | infra | INFO | pool-5-thread-7 | [node_ready_functions:_disable_tls:190] 172.23.108.25: ['Turned off encryption for node: http://172.23.105.187:8091\n', 'Turned off encryption for node: http://172.23.105.98:8091\n', 'Turned off encryption for node: http://172.23.108.25:8091\n', 'Turned off encryption for node: http://172.23.123.22:8091\n', 'SUCCESS: Switched node-to-node encryption off\n'] 2023-08-22 22:04:25,835 | infra | DEBUG | pool-5-thread-7 | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:04:25,835 | infra | INFO | pool-5-thread-7 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:[ip:172.23.108.25 port:8091 ssh_username:root] Kwds:{}' completed 2023-08-22 22:04:25,842 | infra | INFO | pool-5-thread-8 | [node_ready_functions:_disable_tls:190] 172.23.105.98: ['Turned off encryption for node: http://172.23.105.187:8091\n', 'Turned off encryption for node: http://172.23.105.98:8091\n', 'Turned off encryption for node: http://172.23.108.25:8091\n', 'Turned off encryption for node: http://172.23.123.22:8091\n', 'SUCCESS: Switched node-to-node encryption off\n'] 2023-08-22 22:04:25,845 | infra | DEBUG | pool-5-thread-8 | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:04:25,845 | infra | INFO | pool-5-thread-8 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:[ip:172.23.105.98 port:8091 ssh_username:root] Kwds:{}' completed 2023-08-22 22:04:25,851 | infra | INFO | pool-5-thread-5 | [node_ready_functions:_disable_tls:190] 172.23.105.187: ['Turned off encryption for node: http://172.23.105.187:8091\n', 'Turned off encryption for node: http://172.23.105.98:8091\n', 'Turned off encryption for node: http://172.23.108.25:8091\n', 'Turned off encryption for node: http://172.23.123.22:8091\n', 'SUCCESS: Switched node-to-node encryption off\n'] 2023-08-22 22:04:25,852 | infra | DEBUG | pool-5-thread-5 | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:04:25,854 | infra | INFO | pool-5-thread-5 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:[ip:172.23.105.187 port:8091 ssh_username:root] Kwds:{}' completed 2023-08-22 22:04:25,855 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:[ip:172.23.123.22 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:25,855 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:[ip:172.23.108.25 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:25,857 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:[ip:172.23.105.98 port:8091 ssh_username:root] Kwds:{} 2023-08-22 22:04:25,858 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= RollbackTests setup started for test #1 test_rollback_n_times ========= 2023-08-22 22:04:26,269 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:04:26,361 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:04:26,362 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup started for test #1 test_rollback_n_times ========= 2023-08-22 22:04:26,365 | test | DEBUG | MainThread | [basetestcase:setUp:374] Cleaning up cluster 2023-08-22 22:04:26,676 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:4208] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 6186276, u'itemCount': 0, u'quotaPercentUsed': 0.8817595582539923, u'storageTotals': {u'hdd': {u'usedByData': 55595592, u'used': 18657677474L, u'quotaTotal': 211935854592L, u'total': 211935854592L, u'free': 20619197156L}, u'ram': {u'usedByData': 112467360, u'used': 14904541184L, u'quotaTotal': 12754878464L, u'total': 16677388288L, u'quotaUsed': 12754878464L, u'quotaUsedPerNode': 3188719616L, u'quotaTotalPerNode': 3188719616L}}, u'diskUsed': 55595592, u'vbActiveNumNonResident': 0, u'memUsed': 112467360} 2023-08-22 22:04:26,677 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1712] 172.23.105.187 - Remove bucket bucket-3 2023-08-22 22:04:26,678 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1640] Deleting existing bucket bucket-3 on ip:172.23.105.187 port:8091 ssh_username:root 2023-08-22 22:04:26,882 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:4208] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 6186276, u'itemCount': 0, u'quotaPercentUsed': 0.8817595582539923, u'storageTotals': {u'hdd': {u'usedByData': 55595592, u'used': 18657677474L, u'quotaTotal': 211935854592L, u'total': 211935854592L, u'free': 20619197156L}, u'ram': {u'usedByData': 112467360, u'used': 14904541184L, u'quotaTotal': 12754878464L, u'total': 16677388288L, u'quotaUsed': 12754878464L, u'quotaUsedPerNode': 3188719616L, u'quotaTotalPerNode': 3188719616L}}, u'diskUsed': 55595592, u'vbActiveNumNonResident': 0, u'memUsed': 112467360} 2023-08-22 22:04:27,915 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1660] Deleted bucket: bucket-3 from 172.23.105.187 2023-08-22 22:04:27,917 | test | DEBUG | MainThread | [bucket_ready_functions:wait_for_bucket_deletion:1681] Waiting for bucket bucket-3 deletion to finish 2023-08-22 22:04:28,368 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:775] Checking if ns_server is running with timeout 120secs 2023-08-22 22:04:28,490 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:230] Rebalancing all nodes in order to remove nodes 2023-08-22 22:04:28,516 | test | DEBUG | MainThread | [cluster_ready_functions:remove_nodes:203] Ejecting nodes '[u'ns_1@172.23.123.22', u'ns_1@172.23.108.25', u'ns_1@172.23.105.98']' from cluster 2023-08-22 22:04:28,516 | test | DEBUG | MainThread | [rest_client:rebalance:1351] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.105.187,ns_1@172.23.123.22,ns_1@172.23.108.25,ns_1@172.23.105.98', 'ejectedNodes': u'ns_1@172.23.123.22,ns_1@172.23.108.25,ns_1@172.23.105.98', 'user': 'Administrator'} 2023-08-22 22:04:28,602 | test | DEBUG | MainThread | [rest_client:rebalance:1356] Rebalance operation started 2023-08-22 22:04:28,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1590] Rebalance percentage: 0.00 % 2023-08-22 22:04:29,229 | infra | DEBUG | MainThread | [common_lib:sleep:18] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1498 2023-08-22 22:04:29,230 | infra | DEBUG | MainThread | [common_lib:sleep:25] Sleep 3 seconds. Reason: None 2023-08-22 22:04:32,280 | infra | DEBUG | MainThread | [common_lib:sleep:18] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1594 2023-08-22 22:04:32,282 | infra | DEBUG | MainThread | [common_lib:sleep:25] Sleep 5 seconds. Reason: None 2023-08-22 22:04:37,325 | infra | DEBUG | MainThread | [common_lib:sleep:18] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1498 2023-08-22 22:04:37,325 | infra | DEBUG | MainThread | [common_lib:sleep:25] Sleep 3 seconds. Reason: None 2023-08-22 22:04:40,355 | test | INFO | MainThread | [rest_client:monitorRebalance:1507] Rebalance done. Taken 11.7489998341 seconds to complete 2023-08-22 22:04:40,375 | test | DEBUG | MainThread | [common_lib:sleep:18] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1510 2023-08-22 22:04:40,375 | test | INFO | MainThread | [common_lib:sleep:23] Sleep 5 seconds. Reason: Wait after rebalance complete 2023-08-22 22:04:45,592 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:279] Removed all the nodes from cluster associated with ip:172.23.105.187 port:8091 ssh_username:root ? [(u'ns_1@172.23.123.22', 8091), (u'ns_1@172.23.108.25', 8091), (u'ns_1@172.23.105.98', 8091)] 2023-08-22 22:04:45,595 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:(ip:172.23.105.187 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,595 | infra | INFO | pool-5-thread-9 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:(ip:172.23.105.187 port:8091 ssh_username:root, 360) Kwds:{}' started 2023-08-22 22:04:45,595 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:(ip:172.23.123.22 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,596 | infra | INFO | pool-5-thread-10 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:(ip:172.23.123.22 port:8091 ssh_username:root, 360) Kwds:{}' started 2023-08-22 22:04:45,596 | test | DEBUG | pool-5-thread-9 | [cluster_ready_functions:_wait_for_ns_servers:188] Waiting for ns_server @ 172.23.105.187:8091 2023-08-22 22:04:45,598 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:(ip:172.23.108.25 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,598 | infra | INFO | pool-5-thread-11 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:(ip:172.23.108.25 port:8091 ssh_username:root, 360) Kwds:{}' started 2023-08-22 22:04:45,598 | test | DEBUG | pool-5-thread-10 | [cluster_ready_functions:_wait_for_ns_servers:188] Waiting for ns_server @ 172.23.123.22:8091 2023-08-22 22:04:45,599 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: FunctionCallTask: function:> Args:(ip:172.23.105.98 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,599 | infra | INFO | pool-5-thread-12 | [task:start_task:97] Thread 'FunctionCallTask: function:> Args:(ip:172.23.105.98 port:8091 ssh_username:root, 360) Kwds:{}' started 2023-08-22 22:04:45,599 | test | DEBUG | pool-5-thread-11 | [cluster_ready_functions:_wait_for_ns_servers:188] Waiting for ns_server @ 172.23.108.25:8091 2023-08-22 22:04:45,601 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:(ip:172.23.105.187 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,601 | test | DEBUG | pool-5-thread-12 | [cluster_ready_functions:_wait_for_ns_servers:188] Waiting for ns_server @ 172.23.105.98:8091 2023-08-22 22:04:45,638 | test | DEBUG | pool-5-thread-11 | [rest_client:is_ns_server_running:775] Checking if ns_server is running with timeout 360secs 2023-08-22 22:04:45,638 | test | DEBUG | pool-5-thread-10 | [rest_client:is_ns_server_running:775] Checking if ns_server is running with timeout 360secs 2023-08-22 22:04:45,638 | test | DEBUG | pool-5-thread-12 | [rest_client:is_ns_server_running:775] Checking if ns_server is running with timeout 360secs 2023-08-22 22:04:45,638 | test | DEBUG | pool-5-thread-9 | [rest_client:is_ns_server_running:775] Checking if ns_server is running with timeout 360secs 2023-08-22 22:04:45,668 | test | DEBUG | pool-5-thread-12 | [cluster_ready_functions:_wait_for_ns_servers:191] ns_server @ 172.23.105.98:8091 is running 2023-08-22 22:04:45,670 | test | DEBUG | pool-5-thread-11 | [cluster_ready_functions:_wait_for_ns_servers:191] ns_server @ 172.23.108.25:8091 is running 2023-08-22 22:04:45,668 | test | DEBUG | pool-5-thread-10 | [cluster_ready_functions:_wait_for_ns_servers:191] ns_server @ 172.23.123.22:8091 is running 2023-08-22 22:04:45,670 | infra | INFO | pool-5-thread-12 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:(ip:172.23.105.98 port:8091 ssh_username:root, 360) Kwds:{}' completed 2023-08-22 22:04:45,670 | infra | INFO | pool-5-thread-11 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:(ip:172.23.108.25 port:8091 ssh_username:root, 360) Kwds:{}' completed 2023-08-22 22:04:45,671 | infra | INFO | pool-5-thread-10 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:(ip:172.23.123.22 port:8091 ssh_username:root, 360) Kwds:{}' completed 2023-08-22 22:04:45,673 | test | DEBUG | pool-5-thread-9 | [cluster_ready_functions:_wait_for_ns_servers:191] ns_server @ 172.23.105.187:8091 is running 2023-08-22 22:04:45,674 | infra | INFO | pool-5-thread-9 | [task:complete_task:112] Thread 'FunctionCallTask: function:> Args:(ip:172.23.105.187 port:8091 ssh_username:root, 360) Kwds:{}' completed 2023-08-22 22:04:45,674 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:(ip:172.23.123.22 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,674 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:(ip:172.23.108.25 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,674 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for FunctionCallTask: function:> Args:(ip:172.23.105.98 port:8091 ssh_username:root, 360) Kwds:{} 2023-08-22 22:04:45,759 | infra | INFO | MainThread | [basetestcase:tearDownEverything:654] ========== tasks in thread pool ========== 2023-08-22 22:04:45,760 | infra | INFO | MainThread | [basetestcase:tearDownEverything:656] ========================================== 2023-08-22 22:04:45,834 | test | DEBUG | MainThread | [rest_client:set_internalSetting:2299] Update internal setting magmaMinMemoryQuota=256 2023-08-22 22:04:45,834 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= BaseTestCase setup finished for test #1 test_rollback_n_times ========= 2023-08-22 22:04:45,888 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup started for test #1 test_rollback_n_times ========= 2023-08-22 22:04:46,009 | infra | DEBUG | MainThread | [task:__init__:256] Last known rebalance status_id: 75949a1cac8e72bf93c2febf4aee8a4f 2023-08-22 22:04:46,012 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: Rebalance_task_IN=[172.23.123.22,172.23.108.25,172.23.105.98]_OUT=[]_1 2023-08-22 22:04:46,012 | infra | INFO | pool-5-thread-13 | [task:start_task:97] Thread 'Rebalance_task_IN=[172.23.123.22,172.23.108.25,172.23.105.98]_OUT=[]_1' started 2023-08-22 22:04:46,013 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for Rebalance_task_IN=[172.23.123.22,172.23.108.25,172.23.105.98]_OUT=[]_1 2023-08-22 22:04:46,013 | test | DEBUG | pool-5-thread-13 | [rest_client:add_node:1096] Adding remote node 172.23.123.22:8091 to cluster 172.23.105.187:8091 2023-08-22 22:04:48,789 | test | DEBUG | pool-5-thread-13 | [rest_client:add_node:1096] Adding remote node 172.23.108.25:8091 to cluster 172.23.105.187:8091 2023-08-22 22:04:52,358 | test | DEBUG | pool-5-thread-13 | [rest_client:add_node:1096] Adding remote node 172.23.105.98:8091 to cluster 172.23.105.187:8091 2023-08-22 22:04:55,960 | test | DEBUG | pool-5-thread-13 | [rest_client:rebalance:1351] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.105.187,ns_1@172.23.123.22,ns_1@172.23.108.25,ns_1@172.23.105.98', 'ejectedNodes': '', 'user': 'Administrator'} 2023-08-22 22:04:56,038 | test | DEBUG | pool-5-thread-13 | [rest_client:rebalance:1356] Rebalance operation started 2023-08-22 22:04:56,039 | test | INFO | pool-5-thread-13 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | Nodes | Services | Version | CPU | Status | Membership / Recovery | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ | 172.23.105.187 | kv | 7.2.1-5919-enterprise | 1.28799724101 | Cluster node | active / none | | 172.23.123.22 | None | | | <--- IN --- | | | 172.23.108.25 | None | | | <--- IN --- | | | 172.23.105.98 | None | | | <--- IN --- | | +----------------+----------+-----------------------+---------------+--------------+-----------------------+ 2023-08-22 22:04:56,065 | infra | DEBUG | pool-5-thread-13 | [task:call:323] New rebalance status_id: cd868390efc5448fe3cc42b9b6c2b0b7 2023-08-22 22:04:56,088 | test | DEBUG | pool-5-thread-13 | [rest_client:_rebalance_status_and_progress:1590] Rebalance percentage: 0.00 % 2023-08-22 22:04:56,089 | test | INFO | pool-5-thread-13 | [task:check:421] Rebalance - status: running, progress: 0.0 2023-08-22 22:04:56,117 | infra | DEBUG | pool-5-thread-13 | [task:check:448] Wait before next rebalance progress check 2023-08-22 22:04:56,125 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> check():L449 2023-08-22 22:04:56,125 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:25] Sleep 10 seconds. Reason: None 2023-08-22 22:05:06,158 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:18] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1594 2023-08-22 22:05:06,160 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:25] Sleep 5 seconds. Reason: None 2023-08-22 22:05:11,187 | test | INFO | pool-5-thread-13 | [task:check:421] Rebalance - status: none, progress: 100 2023-08-22 22:05:11,219 | test | INFO | pool-5-thread-13 | [task:check:480] Rebalance completed with progress: 100% in 15.1800000668 sec 2023-08-22 22:05:11,220 | infra | INFO | pool-5-thread-13 | [task:complete_task:112] Thread 'Rebalance_task_IN=[172.23.123.22,172.23.108.25,172.23.105.98]_OUT=[]_1' completed 2023-08-22 22:05:11,220 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:4466] **** Add built-in 'cbadminbucket' user to node 172.23.105.187 **** 2023-08-22 22:05:11,430 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:4470] **** Add 'admin' role to 'cbadminbucket' user **** 2023-08-22 22:05:11,530 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.187 | kv | 4.44229694509 | 3.83 GiB | 3.01 GiB | 44.25 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.123.22 | kv | 0 | 4.03 GiB | 3.37 GiB | 45.50 MiB / 4.24 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.108.25 | kv | 0 | 3.83 GiB | 3.20 GiB | 33.00 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.105.98 | kv | 0 | 3.83 GiB | 3.18 GiB | 37.00 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2023-08-22 22:05:11,532 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= ClusterSetup setup complete for test #1 test_rollback_n_times ========= 2023-08-22 22:05:11,532 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup started for test #1 test_rollback_n_times ========= 2023-08-22 22:05:11,555 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:2002] settings/autoFailover params: failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=120&enabled=false&canAbortRebalance=false&failoverPreserveDurabilityMajority=false 2023-08-22 22:05:11,599 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:4466] **** Add built-in 'cbadminbucket' user to node 172.23.105.187 **** 2023-08-22 22:05:11,795 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:4470] **** Add 'admin' role to 'cbadminbucket' user **** 2023-08-22 22:05:11,864 | test | INFO | MainThread | [collections_base:deploy_buckets_from_spec_file:470] Creating bucket from spec: single_bucket.three_scopes_two_collections 2023-08-22 22:05:11,864 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1979] Creating required buckets from template 2023-08-22 22:05:11,963 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:12,013 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: Bucket_create_task_bucket-0 2023-08-22 22:05:12,013 | infra | INFO | pool-5-thread-14 | [task:start_task:97] Thread 'Bucket_create_task_bucket-0' started 2023-08-22 22:05:12,013 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for Bucket_create_task_bucket-0 2023-08-22 22:05:12,063 | infra | INFO | pool-5-thread-14 | [BucketOperations_Rest:create_bucket:357] Creating 'couchbase' bucket bucket-0 2023-08-22 22:05:12,063 | infra | DEBUG | pool-5-thread-14 | [BucketOperations_Rest:create_bucket:359] http://172.23.105.187:8091/pools/default/buckets with param: storageBackend=couchstore&viewFragmentationThreshold%255Bpercentage%255D=50&replicaIndex=1&maxTTL=0&indexCompactionMode=circular&databaseFragmentationThreshold%255Bpercentage%255D=50&flushEnabled=0¶llelDBAndViewCompaction=false&durabilityMinLevel=none&compressionMode=passive&purgeInterval=3&bucketType=couchbase&autoCompactionDefined=false&conflictResolutionType=seqno&name=bucket-0&replicaNumber=3&ramQuotaMB=3041&evictionPolicy=valueOnly 2023-08-22 22:05:12,125 | infra | DEBUG | pool-5-thread-14 | [BucketOperations_Rest:create_bucket:367] 0.06 seconds to create bucket bucket-0 2023-08-22 22:05:12,128 | test | DEBUG | Thread-1 | [task:create_scope_from_spec:3880] Creating scope for 'bucket-0' - _default 2023-08-22 22:05:12,130 | test | DEBUG | Thread-2 | [task:create_collection_from_spec:3909] Creating collection for 'bucket-0:_default' - collection-0 2023-08-22 22:05:12,190 | test | DEBUG | Thread-3 | [task:create_scope_from_spec:3880] Creating scope for 'bucket-0' - scope-1 2023-08-22 22:05:12,252 | test | DEBUG | Thread-4 | [task:create_collection_from_spec:3909] Creating collection for 'bucket-0:scope-1' - collection-3 2023-08-22 22:05:12,313 | test | DEBUG | Thread-5 | [task:create_collection_from_spec:3909] Creating collection for 'bucket-0:scope-1' - collection-4 2023-08-22 22:05:12,377 | test | DEBUG | Thread-6 | [task:create_scope_from_spec:3880] Creating scope for 'bucket-0' - scope-0 2023-08-22 22:05:12,438 | test | DEBUG | Thread-7 | [task:create_collection_from_spec:3909] Creating collection for 'bucket-0:scope-0' - collection-1 2023-08-22 22:05:12,494 | test | DEBUG | Thread-8 | [task:create_collection_from_spec:3909] Creating collection for 'bucket-0:scope-0' - collection-2 2023-08-22 22:05:12,561 | infra | INFO | pool-5-thread-14 | [task:complete_task:112] Thread 'Bucket_create_task_bucket-0' completed 2023-08-22 22:05:12,622 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:4208] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 0, u'itemCount': 0, u'quotaPercentUsed': 0, u'storageTotals': {u'hdd': {u'usedByData': 0, u'used': 18657677474L, u'quotaTotal': 211935854592L, u'total': 211935854592L, u'free': 20619197156L}, u'ram': {u'usedByData': 0, u'used': 14904541184L, u'quotaTotal': 12754878464L, u'total': 16677388288L, u'quotaUsed': 12754878464L, u'quotaUsedPerNode': 3188719616L, u'quotaTotalPerNode': 3188719616L}}, u'diskUsed': 0, u'vbActiveNumNonResident': 0, u'memUsed': 0} 2023-08-22 22:05:12,625 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2741] Waiting for all collections to be created 2023-08-22 22:05:12,720 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.187 | kv | 2.27010533121 | 3.83 GiB | 3.01 GiB | 44.25 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.123.22 | kv | 0.553175660271 | 4.03 GiB | 3.38 GiB | 45.50 MiB / 4.24 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.108.25 | kv | 1.53095702234 | 3.83 GiB | 3.22 GiB | 33.00 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.105.98 | kv | 0 | 3.83 GiB | 3.18 GiB | 37.00 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2023-08-22 22:05:12,795 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:4208] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 0, u'itemCount': 0, u'quotaPercentUsed': 0, u'storageTotals': {u'hdd': {u'usedByData': 0, u'used': 18657677474L, u'quotaTotal': 211935854592L, u'total': 211935854592L, u'free': 20619197156L}, u'ram': {u'usedByData': 0, u'used': 14904541184L, u'quotaTotal': 12754878464L, u'total': 16677388288L, u'quotaUsed': 12754878464L, u'quotaUsedPerNode': 3188719616L, u'quotaTotalPerNode': 3188719616L}}, u'diskUsed': 0, u'vbActiveNumNonResident': 0, u'memUsed': 0} 2023-08-22 22:05:12,924 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +----------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ | bucket-0 | couchbase | couchstore | 3 | none | 0 | 0 | 11.88 GiB | 0.0 Byte | 0.0 Byte | 100 | +----------+-----------+-----------------+----------+------------+-----+-------+-----------+----------+-----------+-----+ 2023-08-22 22:05:12,926 | test | INFO | MainThread | [collections_base:create_clients_for_sdk_pool:491] Creating required SDK clients for client_pool 2023-08-22 22:05:12,927 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:248] Creating SDK connection for 'bucket-0' 2023-08-22 22:05:19,378 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:248] Creating SDK connection for 'bucket-0' 2023-08-22 22:05:19,882 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:248] Creating SDK connection for 'bucket-0' 2023-08-22 22:05:20,220 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:248] Creating SDK connection for 'bucket-0' 2023-08-22 22:05:20,552 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:248] Creating SDK connection for 'bucket-0' 2023-08-22 22:05:21,002 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:248] Creating SDK connection for 'bucket-0' 2023-08-22 22:05:21,331 | test | INFO | MainThread | [collections_base:load_data_from_spec_file:501] Loading data using spec: initial_load 2023-08-22 22:05:21,348 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5211] Performing scope/collection specific operations 2023-08-22 22:05:21,358 | test | DEBUG | MainThread | [bucket_ready_functions:perform_scope_operation:5104] Recreating scopes 2023-08-22 22:05:21,359 | test | DEBUG | MainThread | [bucket_ready_functions:perform_collection_operation:5190] Recreating collections 2023-08-22 22:05:21,361 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5301] Done Performing scope/collection specific operations 2023-08-22 22:05:21,365 | infra | DEBUG | MainThread | [cluster:async_load_gen_docs_from_spec:106] Initializing mutation task for given spec 2023-08-22 22:05:21,367 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: MutateDocsFromSpecTask_2 2023-08-22 22:05:21,368 | infra | INFO | pool-5-thread-15 | [task:start_task:97] Thread 'MutateDocsFromSpecTask_2' started 2023-08-22 22:05:21,368 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for MutateDocsFromSpecTask_2 2023-08-22 22:05:21,634 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:21] Added new task: PrintBucketStats_bucket-0_15 2023-08-22 22:05:21,635 | infra | INFO | pool-5-thread-16 | [task:start_task:97] Thread 'PrintBucketStats_bucket-0_15' started 2023-08-22 22:05:21,637 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6 2023-08-22 22:05:21,637 | infra | INFO | pool-5-thread-17 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6' started 2023-08-22 22:05:21,638 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5 2023-08-22 22:05:21,638 | infra | INFO | pool-5-thread-18 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5' started 2023-08-22 22:05:21,638 | infra | DEBUG | pool-5-thread-17 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:21,638 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9 2023-08-22 22:05:21,640 | infra | DEBUG | pool-5-thread-18 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:21,644 | infra | INFO | pool-5-thread-19 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9' started 2023-08-22 22:05:21,645 | infra | INFO | pool-5-thread-20 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10' started 2023-08-22 22:05:21,644 | infra | DEBUG | pool-5-thread-17 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6 - Iteration 1 2023-08-22 22:05:21,645 | infra | DEBUG | pool-5-thread-18 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5 - Iteration 1 2023-08-22 22:05:21,645 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10 2023-08-22 22:05:21,647 | infra | DEBUG | pool-5-thread-19 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:21,647 | infra | DEBUG | pool-5-thread-20 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:21,650 | infra | DEBUG | pool-5-thread-19 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9 - Iteration 1 2023-08-22 22:05:21,650 | infra | DEBUG | pool-5-thread-20 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10 - Iteration 1 2023-08-22 22:05:21,653 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12 2023-08-22 22:05:21,654 | infra | INFO | pool-5-thread-21 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12' started 2023-08-22 22:05:21,654 | infra | DEBUG | pool-5-thread-21 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:21,655 | infra | DEBUG | pool-5-thread-21 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12 - Iteration 1 2023-08-22 22:05:21,654 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14 2023-08-22 22:05:21,657 | infra | INFO | pool-5-thread-22 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14' started 2023-08-22 22:05:21,657 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6 2023-08-22 22:05:21,657 | infra | DEBUG | pool-5-thread-22 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:21,657 | infra | DEBUG | pool-5-thread-22 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14 - Iteration 1 2023-08-22 22:05:21,798 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> call():L3701 2023-08-22 22:05:21,799 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:25] Sleep 1 seconds. Reason: None 2023-08-22 22:05:22,940 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3656] Ops rate for 'bucket-0': 359.400000 2023-08-22 22:05:22,946 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> call():L3701 2023-08-22 22:05:22,947 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:25] Sleep 1 seconds. Reason: None 2023-08-22 22:05:23,661 | infra | INFO | pool-5-thread-20 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10' completed 2023-08-22 22:05:23,677 | infra | INFO | pool-5-thread-22 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14' completed 2023-08-22 22:05:23,681 | infra | INFO | pool-5-thread-19 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9' completed 2023-08-22 22:05:23,681 | infra | INFO | pool-5-thread-18 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5' completed 2023-08-22 22:05:23,681 | infra | INFO | pool-5-thread-17 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6' completed 2023-08-22 22:05:23,684 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6 are 10000 2023-08-22 22:05:23,684 | test | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 0 to 10000 of thread_name LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6 2023-08-22 22:05:23,684 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5 2023-08-22 22:05:23,684 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5 are 10000 2023-08-22 22:05:23,686 | test | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 0 to 10000 of thread_name LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5 2023-08-22 22:05:23,686 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9 2023-08-22 22:05:23,687 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9 are 10000 2023-08-22 22:05:23,687 | test | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 0 to 10000 of thread_name LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9 2023-08-22 22:05:23,687 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10 2023-08-22 22:05:23,687 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10 are 10000 2023-08-22 22:05:23,688 | test | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 0 to 10000 of thread_name LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10 2023-08-22 22:05:23,688 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12 2023-08-22 22:05:23,694 | infra | INFO | pool-5-thread-21 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12' completed 2023-08-22 22:05:23,696 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12 are 10000 2023-08-22 22:05:23,696 | test | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 0 to 10000 of thread_name LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12 2023-08-22 22:05:23,697 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14 2023-08-22 22:05:23,697 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14 are 10000 2023-08-22 22:05:23,697 | test | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 0 to 10000 of thread_name LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14 2023-08-22 22:05:23,697 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4038] ========= Tasks in loadgen pool======= 2023-08-22 22:05:23,697 | infra | WARNING | pool-5-thread-15 | [task_manager:print_tasks_in_pool:82] Task 'PrintBucketStats_bucket-0_15' not completed 2023-08-22 22:05:23,698 | infra | WARNING | pool-5-thread-15 | [task_manager:print_tasks_in_pool:82] Task 'MutateDocsFromSpecTask_2' not completed 2023-08-22 22:05:23,698 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4040] ====================================== 2023-08-22 22:05:23,700 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default_collection-0_create_ttl=0_create__0_10000_6' complete. Loaded 10000 items 2023-08-22 22:05:23,700 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0__default__default_create_ttl=0_create__0_10000_5' complete. Loaded 10000 items 2023-08-22 22:05:23,700 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-3_create_ttl=0_create__0_10000_9' complete. Loaded 10000 items 2023-08-22 22:05:23,700 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-1_collection-4_create_ttl=0_create__0_10000_10' complete. Loaded 10000 items 2023-08-22 22:05:23,701 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-1_create_ttl=0_create__0_10000_12' complete. Loaded 10000 items 2023-08-22 22:05:23,701 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_2_bucket-0_scope-0_collection-2_create_ttl=0_create__0_10000_14' complete. Loaded 10000 items 2023-08-22 22:05:23,703 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4038] ========= Tasks in loadgen pool======= 2023-08-22 22:05:23,703 | infra | WARNING | pool-5-thread-15 | [task_manager:print_tasks_in_pool:82] Task 'PrintBucketStats_bucket-0_15' not completed 2023-08-22 22:05:23,703 | infra | WARNING | pool-5-thread-15 | [task_manager:print_tasks_in_pool:82] Task 'MutateDocsFromSpecTask_2' not completed 2023-08-22 22:05:23,703 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4040] ====================================== 2023-08-22 22:05:23,703 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4038] ========= Tasks in loadgen pool======= 2023-08-22 22:05:23,704 | infra | WARNING | pool-5-thread-15 | [task_manager:print_tasks_in_pool:82] Task 'PrintBucketStats_bucket-0_15' not completed 2023-08-22 22:05:23,706 | infra | WARNING | pool-5-thread-15 | [task_manager:print_tasks_in_pool:82] Task 'MutateDocsFromSpecTask_2' not completed 2023-08-22 22:05:23,706 | infra | DEBUG | pool-5-thread-15 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4040] ====================================== 2023-08-22 22:05:23,707 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:24] Getting task result for PrintBucketStats_bucket-0_15 2023-08-22 22:05:23,947 | test | INFO | pool-5-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket-0' +---------+------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------+---------+ | 359.400 | ...................................................................................................X | 359.400 | +---------+------------------------------------------------------------------------------------------------------+---------+ 2023-08-22 22:05:23,950 | infra | INFO | pool-5-thread-16 | [task:complete_task:112] Thread 'PrintBucketStats_bucket-0_15' completed 2023-08-22 22:05:23,950 | infra | INFO | pool-5-thread-15 | [task:complete_task:112] Thread 'MutateDocsFromSpecTask_2' completed 2023-08-22 22:05:24,023 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.187 | kv | 2.27010533121 | 3.83 GiB | 3.01 GiB | 44.25 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.123.22 | kv | 0.553175660271 | 4.03 GiB | 3.38 GiB | 45.50 MiB / 4.24 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.108.25 | kv | 1.53095702234 | 3.83 GiB | 3.22 GiB | 33.00 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.105.98 | kv | 6.55979208666 | 3.83 GiB | 3.17 GiB | 37.00 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2023-08-22 22:05:24,075 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:24,082 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:24,144 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:24,151 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_16 2023-08-22 22:05:24,151 | infra | INFO | pool-5-thread-23 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_16' started 2023-08-22 22:05:24,151 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:24,157 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:24,200 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:24,203 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_17 2023-08-22 22:05:24,203 | infra | INFO | pool-5-thread-24 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_17' started 2023-08-22 22:05:24,204 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:24,210 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:24,289 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:24,290 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_18 2023-08-22 22:05:24,292 | infra | INFO | pool-5-thread-25 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_18' started 2023-08-22 22:05:24,293 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:24,299 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:24,358 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:24,359 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_19 2023-08-22 22:05:24,361 | infra | INFO | pool-5-thread-26 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_19' started 2023-08-22 22:05:24,361 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_num_items_for_persistence_16 2023-08-22 22:05:24,362 | infra | DEBUG | pool-5-thread-26 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:25,788 | test | DEBUG | pool-5-thread-24 | [task:_compare:2807] Comparing 0 == 0 2023-08-22 22:05:25,789 | test | DEBUG | pool-5-thread-24 | [task:_get_checkpoint_stats_and_compare:2796] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.123.22': 0} for bucket 'bucket-0' 2023-08-22 22:05:25,789 | infra | INFO | pool-5-thread-24 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_17' completed 2023-08-22 22:05:26,046 | test | DEBUG | pool-5-thread-23 | [task:_compare:2807] Comparing 0 == 0 2023-08-22 22:05:26,046 | test | DEBUG | pool-5-thread-23 | [task:_get_checkpoint_stats_and_compare:2796] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.105.187': 0} for bucket 'bucket-0' 2023-08-22 22:05:26,046 | infra | INFO | pool-5-thread-23 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_16' completed 2023-08-22 22:05:26,048 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:05:26,049 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_num_items_for_persistence_17 2023-08-22 22:05:26,049 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.123.22 2023-08-22 22:05:26,051 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_num_items_for_persistence_18 2023-08-22 22:05:26,098 | test | DEBUG | pool-5-thread-25 | [task:_compare:2807] Comparing 0 == 0 2023-08-22 22:05:26,098 | test | DEBUG | pool-5-thread-25 | [task:_get_checkpoint_stats_and_compare:2796] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.108.25': 0} for bucket 'bucket-0' 2023-08-22 22:05:26,098 | infra | INFO | pool-5-thread-25 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_18' completed 2023-08-22 22:05:26,099 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:05:26,099 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_num_items_for_persistence_19 2023-08-22 22:05:26,177 | test | DEBUG | pool-5-thread-26 | [task:_compare:2807] Comparing 0 == 0 2023-08-22 22:05:26,178 | test | DEBUG | pool-5-thread-26 | [task:_get_checkpoint_stats_and_compare:2796] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.105.98': 0} for bucket 'bucket-0' 2023-08-22 22:05:26,180 | infra | INFO | pool-5-thread-26 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_19' completed 2023-08-22 22:05:26,180 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:05:26,180 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4954] Validating collection stats and item counts 2023-08-22 22:05:26,246 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:26,250 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:26,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-seqno 2023-08-22 22:05:26,911 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:05:26,917 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:26,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-seqno 2023-08-22 22:05:27,456 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.123.22 2023-08-22 22:05:27,461 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:27,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-seqno 2023-08-22 22:05:28,102 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:05:28,111 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:28,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-seqno 2023-08-22 22:05:28,779 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:05:28,779 | test | DEBUG | MainThread | [bucket_ready_functions:verify_stats_for_bucket:2475] Verifying stats for bucket bucket-0 2023-08-22 22:05:28,846 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:28,911 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:28,917 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:28,993 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:29,059 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:29,114 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:29,163 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:29,240 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:29,334 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:29,414 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:29,479 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for vb_replica_curr_items on bucket bucket-0 2023-08-22 22:05:29,480 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_vb_replica_curr_items_20 2023-08-22 22:05:29,480 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for curr_items_tot on bucket bucket-0 2023-08-22 22:05:29,480 | infra | INFO | pool-5-thread-27 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_vb_replica_curr_items_20' started 2023-08-22 22:05:29,482 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_curr_items_tot_21 2023-08-22 22:05:29,483 | infra | INFO | pool-5-thread-28 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_curr_items_tot_21' started 2023-08-22 22:05:29,483 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_vb_replica_curr_items_20 2023-08-22 22:05:29,483 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:29,483 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:29,651 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:29,677 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:29,776 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:29,796 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:29,957 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:29,963 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 all 2023-08-22 22:05:30,141 | test | DEBUG | pool-5-thread-28 | [task:_compare:2807] Comparing 240000 == 240000 2023-08-22 22:05:30,141 | test | DEBUG | pool-5-thread-27 | [task:_compare:2807] Comparing 180000 == 180000 2023-08-22 22:05:30,141 | test | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2759] Ready: curr_items_tot 240000 == 240000. Received: {'172.23.105.187': '60000', '172.23.105.98': '60000', '172.23.123.22': '60000', '172.23.108.25': '60000'} for bucket 'bucket-0' 2023-08-22 22:05:30,142 | test | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2759] Ready: vb_replica_curr_items 180000 == 180000. Received: {'172.23.105.187': '44994', '172.23.105.98': '45006', '172.23.123.22': '44994', '172.23.108.25': '45006'} for bucket 'bucket-0' 2023-08-22 22:05:30,142 | infra | INFO | pool-5-thread-28 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_curr_items_tot_21' completed 2023-08-22 22:05:30,142 | infra | INFO | pool-5-thread-27 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_vb_replica_curr_items_20' completed 2023-08-22 22:05:30,144 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_curr_items_tot_21 2023-08-22 22:05:30,144 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:05:30,144 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.123.22 2023-08-22 22:05:30,145 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:05:30,145 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:05:30,148 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:05:30,148 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.123.22 2023-08-22 22:05:30,150 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:05:30,151 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:05:30,286 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:30,290 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:30,369 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:30,423 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:30,496 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:30,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 collections 2023-08-22 22:05:30,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 collections 2023-08-22 22:05:30,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 collections 2023-08-22 22:05:30,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 collections 2023-08-22 22:05:31,081 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:05:31,081 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.123.22 2023-08-22 22:05:31,082 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:05:31,082 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:05:31,138 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:31,142 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:31,210 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:31,266 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:31,332 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:31,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 scopes 2023-08-22 22:05:31,542 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.187 2023-08-22 22:05:31,542 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.123.22 2023-08-22 22:05:31,545 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:05:31,545 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:05:31,608 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ | 172.23.105.187 | kv | 39.363304103 | 3.83 GiB | 2.94 GiB | 44.25 MiB / 3.99 GiB | 0 / 0 | 7.2.1-5919-enterprise | | 172.23.123.22 | kv | 13.5344572802 | 4.03 GiB | 3.31 GiB | 45.50 MiB / 4.24 GiB | 15006 / 44994 | 7.2.1-5919-enterprise | | 172.23.108.25 | kv | 23.9261988313 | 3.83 GiB | 3.14 GiB | 33.00 MiB / 3.99 GiB | 10344 / 30869 | 7.2.1-5919-enterprise | | 172.23.105.98 | kv | 34.7778383187 | 3.83 GiB | 3.08 GiB | 37.00 MiB / 3.99 GiB | 3544 / 10211 | 7.2.1-5919-enterprise | +----------------+----------+-----------------+-----------+----------+----------------------+-------------------+-----------------------+ 2023-08-22 22:05:31,684 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:4208] Stats: {u'opsPerSec': 2894.4, u'diskFetches': 0, u'dataUsed': 20544644, u'itemCount': 28894, u'quotaPercentUsed': 1.273617216020538, u'storageTotals': {u'hdd': {u'usedByData': 41838988, u'used': 18657677474L, u'quotaTotal': 211935854592L, u'total': 211935854592L, u'free': 20619197156L}, u'ram': {u'usedByData': 162448328, u'used': 14662639616L, u'quotaTotal': 12754878464L, u'total': 16677388288L, u'quotaUsed': 12754878464L, u'quotaUsedPerNode': 3188719616L, u'quotaTotalPerNode': 3188719616L}}, u'diskUsed': 41838988, u'vbActiveNumNonResident': 0, u'memUsed': 162448328} 2023-08-22 22:05:31,836 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +----------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ | bucket-0 | couchbase | couchstore | 3 | none | 0 | 28894 | 11.88 GiB | 154.92 MiB | 39.90 MiB | 100 | +----------+-----------+-----------------+----------+------------+-----+-------+-----------+------------+-----------+-----+ 2023-08-22 22:05:31,838 | test | INFO | MainThread | [basetestcase:log_setup_status:683] ========= CollectionBase setup complete for test #1 test_rollback_n_times ========= 2023-08-22 22:05:31,864 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:2002] settings/autoFailover params: failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=120&enabled=false&canAbortRebalance=false&failoverPreserveDurabilityMajority=false 2023-08-22 22:05:31,930 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:641] All nodes in cluster: [ip:172.23.105.187 port:8091 ssh_username:root, ip:172.23.123.22 port:8091 ssh_username:root, ip:172.23.108.25 port:8091 ssh_username:root, ip:172.23.105.98 port:8091 ssh_username:root] 2023-08-22 22:05:31,934 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:32,003 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:32,068 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:32,151 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:32,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-details 2023-08-22 22:05:35,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-details 2023-08-22 22:05:37,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-details 2023-08-22 22:05:40,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-details 2023-08-22 22:05:42,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket 2023-08-22 22:05:42,957 | test | INFO | MainThread | [cb_error:create:39] Simulating 'stop_persistence' in 172.23.105.187 2023-08-22 22:05:42,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbepctl localhost:11210 -u Administrator -p password -b bucket-0 stop 2023-08-22 22:05:43,575 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5211] Performing scope/collection specific operations 2023-08-22 22:05:43,581 | test | DEBUG | MainThread | [bucket_ready_functions:perform_scope_operation:5104] Recreating scopes 2023-08-22 22:05:43,582 | test | DEBUG | MainThread | [bucket_ready_functions:perform_collection_operation:5190] Recreating collections 2023-08-22 22:05:43,582 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:5301] Done Performing scope/collection specific operations 2023-08-22 22:05:46,187 | infra | DEBUG | MainThread | [cluster:async_load_gen_docs_from_spec:106] Initializing mutation task for given spec 2023-08-22 22:05:46,187 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: MutateDocsFromSpecTask_22 2023-08-22 22:05:46,190 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for MutateDocsFromSpecTask_22 2023-08-22 22:05:46,190 | infra | INFO | pool-5-thread-29 | [task:start_task:97] Thread 'MutateDocsFromSpecTask_22' started 2023-08-22 22:05:46,371 | infra | INFO | pool-5-thread-29 | [task_manager:add_new_task:21] Added new task: PrintBucketStats_bucket-0_29 2023-08-22 22:05:46,371 | infra | INFO | pool-5-thread-30 | [task:start_task:97] Thread 'PrintBucketStats_bucket-0_29' started 2023-08-22 22:05:46,371 | infra | INFO | pool-5-thread-29 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25 2023-08-22 22:05:46,371 | infra | INFO | pool-5-thread-2 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25' started 2023-08-22 22:05:46,371 | infra | INFO | pool-5-thread-29 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26 2023-08-22 22:05:46,372 | infra | INFO | pool-5-thread-1 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26' started 2023-08-22 22:05:46,372 | infra | INFO | pool-5-thread-29 | [task_manager:add_new_task:21] Added new task: LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28 2023-08-22 22:05:46,372 | infra | INFO | pool-5-thread-4 | [task:start_task:97] Thread 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28' started 2023-08-22 22:05:46,372 | infra | DEBUG | pool-5-thread-29 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25 2023-08-22 22:05:46,374 | infra | DEBUG | pool-5-thread-4 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:46,374 | infra | DEBUG | pool-5-thread-4 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28 - Iteration 1 2023-08-22 22:05:46,372 | infra | DEBUG | pool-5-thread-2 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:46,372 | infra | DEBUG | pool-5-thread-1 | [task:call:526] Total req iterations: 1 2023-08-22 22:05:46,375 | infra | DEBUG | pool-5-thread-2 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25 - Iteration 1 2023-08-22 22:05:46,375 | infra | DEBUG | pool-5-thread-1 | [task:call:532] LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26 - Iteration 1 2023-08-22 22:05:46,519 | test | DEBUG | pool-5-thread-30 | [task:record_bucket_ops:3656] Ops rate for 'bucket-0': 0.000000 2023-08-22 22:05:46,523 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> call():L3701 2023-08-22 22:05:46,525 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:25] Sleep 1 seconds. Reason: None 2023-08-22 22:05:47,496 | infra | INFO | pool-5-thread-2 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25' completed 2023-08-22 22:05:47,496 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25 are 10000 2023-08-22 22:05:47,497 | test | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 10000 to 20000 of thread_name LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25 2023-08-22 22:05:47,497 | infra | DEBUG | pool-5-thread-29 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26 2023-08-22 22:05:47,510 | infra | INFO | pool-5-thread-1 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26' completed 2023-08-22 22:05:47,512 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26 are 10000 2023-08-22 22:05:47,512 | test | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 10000 to 20000 of thread_name LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26 2023-08-22 22:05:47,513 | infra | DEBUG | pool-5-thread-29 | [task_manager:get_task_result:24] Getting task result for LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28 2023-08-22 22:05:47,519 | infra | INFO | pool-5-thread-4 | [task:complete_task:112] Thread 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28' completed 2023-08-22 22:05:47,519 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4000] Items loaded in task LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28 are 10000 2023-08-22 22:05:47,520 | test | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4029] Failed to load 0 docs from 10000 to 20000 of thread_name LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28 2023-08-22 22:05:47,520 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4038] ========= Tasks in loadgen pool======= 2023-08-22 22:05:47,520 | infra | WARNING | pool-5-thread-29 | [task_manager:print_tasks_in_pool:82] Task 'PrintBucketStats_bucket-0_29' not completed 2023-08-22 22:05:47,522 | infra | WARNING | pool-5-thread-29 | [task_manager:print_tasks_in_pool:82] Task 'MutateDocsFromSpecTask_22' not completed 2023-08-22 22:05:47,522 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4040] ====================================== 2023-08-22 22:05:47,522 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-3_create_ttl=0_create_NONE_10000_20000_25' complete. Loaded 10000 items 2023-08-22 22:05:47,523 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0__default_collection-0_create_ttl=0_create_NONE_10000_20000_26' complete. Loaded 10000 items 2023-08-22 22:05:47,523 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4043] Task 'LoadDocs_MutateDocsFromSpecTask_22_bucket-0_scope-1_collection-4_create_ttl=0_create_NONE_10000_20000_28' complete. Loaded 10000 items 2023-08-22 22:05:47,523 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4038] ========= Tasks in loadgen pool======= 2023-08-22 22:05:47,523 | infra | WARNING | pool-5-thread-29 | [task_manager:print_tasks_in_pool:82] Task 'PrintBucketStats_bucket-0_29' not completed 2023-08-22 22:05:47,523 | infra | WARNING | pool-5-thread-29 | [task_manager:print_tasks_in_pool:82] Task 'MutateDocsFromSpecTask_22' not completed 2023-08-22 22:05:47,523 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4040] ====================================== 2023-08-22 22:05:47,523 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4038] ========= Tasks in loadgen pool======= 2023-08-22 22:05:47,525 | infra | WARNING | pool-5-thread-29 | [task_manager:print_tasks_in_pool:82] Task 'PrintBucketStats_bucket-0_29' not completed 2023-08-22 22:05:47,525 | infra | WARNING | pool-5-thread-29 | [task_manager:print_tasks_in_pool:82] Task 'MutateDocsFromSpecTask_22' not completed 2023-08-22 22:05:47,525 | infra | DEBUG | pool-5-thread-29 | [task:_MutateDocsFromSpecTask__wait_for_task_completion:4040] ====================================== 2023-08-22 22:05:47,526 | infra | DEBUG | pool-5-thread-29 | [task_manager:get_task_result:24] Getting task result for PrintBucketStats_bucket-0_29 2023-08-22 22:05:47,641 | test | DEBUG | pool-5-thread-30 | [task:record_bucket_ops:3656] Ops rate for 'bucket-0': 0.000000 2023-08-22 22:05:47,645 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> call():L3701 2023-08-22 22:05:47,645 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:25] Sleep 1 seconds. Reason: None 2023-08-22 22:05:48,647 | infra | INFO | pool-5-thread-30 | [task:complete_task:112] Thread 'PrintBucketStats_bucket-0_29' completed 2023-08-22 22:05:48,647 | infra | INFO | pool-5-thread-29 | [task:complete_task:112] Thread 'MutateDocsFromSpecTask_22' completed 2023-08-22 22:05:48,651 | test | INFO | MainThread | [rollback_tests:test_rollback_n_times:184] Validating stats 2023-08-22 22:05:48,657 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.98 with username: root, password: couchbase 2023-08-22 22:05:48,720 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:48,720 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_30 2023-08-22 22:05:48,720 | infra | INFO | pool-5-thread-3 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_30' started 2023-08-22 22:05:48,721 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.98: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:48,724 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.108.25 with username: root, password: couchbase 2023-08-22 22:05:48,805 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:48,805 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_31 2023-08-22 22:05:48,805 | infra | INFO | pool-5-thread-6 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_31' started 2023-08-22 22:05:48,805 | infra | DEBUG | pool-5-thread-6 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.108.25: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:48,809 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.105.187 with username: root, password: couchbase 2023-08-22 22:05:48,890 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:48,890 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_32 2023-08-22 22:05:48,890 | infra | INFO | pool-5-thread-7 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_32' started 2023-08-22 22:05:48,891 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:48,894 | infra | DEBUG | MainThread | [remote_util:connect:215] Connecting to 172.23.123.22 with username: root, password: couchbase 2023-08-22 22:05:48,948 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:562] Starting StatsWaitTask for num_items_for_persistence on bucket bucket-0 2023-08-22 22:05:48,950 | infra | INFO | MainThread | [task_manager:add_new_task:21] Added new task: StatsWaitTask_bucket-0_num_items_for_persistence_33 2023-08-22 22:05:48,950 | infra | INFO | pool-5-thread-8 | [task:start_task:97] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_33' started 2023-08-22 22:05:48,950 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_num_items_for_persistence_30 2023-08-22 22:05:48,950 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.123.22: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:50,296 | test | DEBUG | pool-5-thread-3 | [task:_compare:2807] Comparing 0 == 0 2023-08-22 22:05:50,296 | test | DEBUG | pool-5-thread-3 | [task:_get_checkpoint_stats_and_compare:2796] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.105.98': 0} for bucket 'bucket-0' 2023-08-22 22:05:50,296 | infra | INFO | pool-5-thread-3 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_30' completed 2023-08-22 22:05:50,296 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.105.98 2023-08-22 22:05:50,298 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_num_items_for_persistence_31 2023-08-22 22:05:50,313 | test | DEBUG | pool-5-thread-8 | [task:_compare:2807] Comparing 0 == 0 2023-08-22 22:05:50,315 | test | DEBUG | pool-5-thread-8 | [task:_get_checkpoint_stats_and_compare:2796] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.123.22': 0} for bucket 'bucket-0' 2023-08-22 22:05:50,315 | infra | INFO | pool-5-thread-8 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_33' completed 2023-08-22 22:05:50,427 | test | DEBUG | pool-5-thread-6 | [task:_compare:2807] Comparing 0 == 0 2023-08-22 22:05:50,427 | test | DEBUG | pool-5-thread-6 | [task:_get_checkpoint_stats_and_compare:2796] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.108.25': 0} for bucket 'bucket-0' 2023-08-22 22:05:50,427 | infra | INFO | pool-5-thread-6 | [task:complete_task:112] Thread 'StatsWaitTask_bucket-0_num_items_for_persistence_31' completed 2023-08-22 22:05:50,428 | infra | DEBUG | MainThread | [remote_util:disconnect:228] Disconnecting ssh_client for 172.23.108.25 2023-08-22 22:05:50,430 | infra | DEBUG | MainThread | [task_manager:get_task_result:24] Getting task result for StatsWaitTask_bucket-0_num_items_for_persistence_32 2023-08-22 22:05:50,529 | test | DEBUG | pool-5-thread-7 | [task:_compare:2807] Comparing 30256 == 30000 2023-08-22 22:05:50,529 | test | DEBUG | pool-5-thread-7 | [task:_get_checkpoint_stats_and_compare:2789] Not Ready: num_items_for_persistence 30256 == 30000. Received: {'172.23.105.187': 30256} for bucket 'bucket-0' 2023-08-22 22:05:50,529 | infra | DEBUG | pool-5-thread-7 | [task:_get_checkpoint_stats_and_compare:2793] Wait before next StatWaitTask check 2023-08-22 22:05:50,533 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2794 2023-08-22 22:05:50,533 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:25] Sleep 5 seconds. Reason: None 2023-08-22 22:05:55,535 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:05:56,926 | test | DEBUG | pool-5-thread-7 | [task:_compare:2807] Comparing 30256 == 30000 2023-08-22 22:05:56,927 | test | DEBUG | pool-5-thread-7 | [task:_get_checkpoint_stats_and_compare:2789] Not Ready: num_items_for_persistence 30256 == 30000. Received: {'172.23.105.187': 30256} for bucket 'bucket-0' 2023-08-22 22:05:56,927 | infra | DEBUG | pool-5-thread-7 | [task:_get_checkpoint_stats_and_compare:2793] Wait before next StatWaitTask check 2023-08-22 22:05:56,931 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2794 2023-08-22 22:05:56,933 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:25] Sleep 5 seconds. Reason: None 2023-08-22 22:06:01,933 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint 2023-08-22 22:06:03,312 | test | DEBUG | pool-5-thread-7 | [task:_compare:2807] Comparing 30256 == 30000 2023-08-22 22:06:03,313 | test | DEBUG | pool-5-thread-7 | [task:_get_checkpoint_stats_and_compare:2789] Not Ready: num_items_for_persistence 30256 == 30000. Received: {'172.23.105.187': 30256} for bucket 'bucket-0' 2023-08-22 22:06:03,315 | infra | DEBUG | pool-5-thread-7 | [task:_get_checkpoint_stats_and_compare:2793] Wait before next StatWaitTask check 2023-08-22 22:06:03,319 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:18] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2794 2023-08-22 22:06:03,319 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:25] Sleep 5 seconds. Reason: None 2023-08-22 22:06:08,321 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3229] Running command on 172.23.105.187: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 checkpoint Build was aborted Aborted by Ashwin G Archiving artifacts Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion No emails were triggered. Finished: ABORTED