Started by user Sumedh Basarkod [EnvInject] - Loading node environment variables. Building remotely on slv-sc2401-32g-12c (P0 jython_slave) in workspace /data/workspace/rebalance_sumedh_temp [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/rebalance_sumedh_temp # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 9a76e9a73bd44b1b7eba5f4c83306416d2c4ca28 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 9a76e9a73bd44b1b7eba5f4c83306416d2c4ca28 > /usr/bin/git rev-list 5cd10ac5ee2186350ccf68b2f91baa67e626dec6 # timeout=10 [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins3233659538329520544.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:172.23.105.215 port:8091 [_2] ip:172.23.105.217 port:8091 [_3] ip:172.23.105.219 port:8091 [_4] ip:172.23.105.220 port:8091 [_5] ip:172.23.106.237 port:8091 [_6] ip:172.23.107.90 port:8091' + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + git fetch http://review.couchbase.org/TAF refs/changes/88/145788/6 From http://review.couchbase.org/TAF * branch refs/changes/88/145788/6 -> FETCH_HEAD + git cherry-pick FETCH_HEAD [detached HEAD ae639c7] Temp patch to repro MB Author: sumedhpb 1 file changed, 27 insertions(+) [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins6640799947697653330.sh + guides/gradlew --refresh-dependencies install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p version=7.0.0-4465,product=cb,parallel=True' Starting a Gradle Daemon, 34 busy Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p version=7.0.0-4465,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p version=7.0.0-4465,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p version=7.0.0-4465,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p version=7.0.0-4465,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p version=7.0.0-4465,product=cb,parallel=True > Task :install 2021-02-15 02:42:14,403 - root - INFO - Done with uninstall on 172.23.105.220. 2021-02-15 02:42:14,608 - root - INFO - Done with uninstall on 172.23.105.215. 2021-02-15 02:42:14,642 - root - INFO - Done with uninstall on 172.23.105.217. 2021-02-15 02:42:18,796 - root - INFO - Done with uninstall on 172.23.107.90. 2021-02-15 02:42:19,538 - root - INFO - Done with uninstall on 172.23.105.219. 2021-02-15 02:43:36,207 - root - INFO - Done with uninstall on 172.23.106.237. 2021-02-15 02:46:48,423 - root - INFO - Done with install on 172.23.105.217. 2021-02-15 02:46:48,767 - root - INFO - Done with install on 172.23.105.220. 2021-02-15 02:47:00,467 - root - INFO - Done with install on 172.23.105.215. 2021-02-15 02:47:03,450 - root - INFO - Done with install on 172.23.107.90. 2021-02-15 02:47:07,219 - root - INFO - Done with install on 172.23.105.219. 2021-02-15 02:47:50,157 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.217 2021-02-15 02:47:50,174 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.220 2021-02-15 02:47:50,303 - root - INFO - Done with init on 172.23.105.217. 2021-02-15 02:47:50,336 - root - INFO - Done with cleanup on 172.23.105.217. 2021-02-15 02:47:50,349 - root - INFO - Done with init on 172.23.105.220. 2021-02-15 02:47:50,375 - root - INFO - Done with cleanup on 172.23.105.220. 2021-02-15 02:48:01,857 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.215 2021-02-15 02:48:02,006 - root - INFO - Done with init on 172.23.105.215. 2021-02-15 02:48:02,032 - root - INFO - Done with cleanup on 172.23.105.215. 2021-02-15 02:48:04,802 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.107.90 2021-02-15 02:48:04,960 - root - INFO - Done with init on 172.23.107.90. 2021-02-15 02:48:04,996 - root - INFO - Done with cleanup on 172.23.107.90. 2021-02-15 02:48:08,592 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.219 2021-02-15 02:48:08,746 - root - INFO - Done with init on 172.23.105.219. 2021-02-15 02:48:08,772 - root - INFO - Done with cleanup on 172.23.105.219. 2021-02-15 02:48:24,532 - root - INFO - Done with install on 172.23.106.237. 2021-02-15 02:49:25,897 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.237 2021-02-15 02:49:26,055 - root - INFO - Done with init on 172.23.106.237. 2021-02-15 02:49:26,088 - root - INFO - Done with cleanup on 172.23.106.237. 2021-02-15 02:49:40,338 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-15 02:49:40,362 - root - INFO - node:172.23.105.215:8091 version:7.0.0-4465-enterprise aFamily:inet services:[u'kv'] 2021-02-15 02:49:40,388 - root - INFO - node:172.23.105.217:8091 version:7.0.0-4465-enterprise aFamily:inet services:[u'kv'] 2021-02-15 02:49:40,414 - root - INFO - node:172.23.105.219:8091 version:7.0.0-4465-enterprise aFamily:inet services:[u'kv'] 2021-02-15 02:49:40,437 - root - INFO - node:172.23.105.220:8091 version:7.0.0-4465-enterprise aFamily:inet services:[u'kv'] 2021-02-15 02:49:40,459 - root - INFO - node:172.23.106.237:8091 version:7.0.0-4465-enterprise aFamily:inet services:[u'kv'] 2021-02-15 02:49:40,489 - root - INFO - node:172.23.107.90:8091 version:7.0.0-4465-enterprise aFamily:inet services:[u'kv'] 2021-02-15 02:49:40,490 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-15 02:49:40,490 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-15 02:49:40,490 - root - INFO - INSTALL COMPLETED ON: 172.23.105.215 2021-02-15 02:49:40,490 - root - INFO - INSTALL COMPLETED ON: 172.23.105.217 2021-02-15 02:49:40,492 - root - INFO - INSTALL COMPLETED ON: 172.23.105.219 2021-02-15 02:49:40,493 - root - INFO - INSTALL COMPLETED ON: 172.23.105.220 2021-02-15 02:49:40,493 - root - INFO - INSTALL COMPLETED ON: 172.23.106.237 2021-02-15 02:49:40,493 - root - INFO - INSTALL COMPLETED ON: 172.23.107.90 2021-02-15 02:49:40,493 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-15 02:49:40,497 - root - INFO - TOTAL INSTALL TIME = 546.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 9m 20s 1 actionable task: 1 executed + guides/gradlew testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral -m rest' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=True,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral -m rest > Task :compileJava Note: /data/workspace/rebalance_sumedh_temp/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Global Test input params: {'cluster_name': 'durability_volume', 'collect_pcaps': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 6, 'rerun': 'False'} Logs will be stored at /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-15_02-49-48/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini rerun=False,get-cbcollect-info=True,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral' Test Input params: {'cluster_name': 'durability_volume', 'ini': '/tmp/durability_volume.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'nodes_in': '2', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-15_02-49-48/test_1', 'nodes_init': '3', 'collect_pcaps': 'True', 'bucket_spec': 'multi_bucket.buckets_all_ephemeral', 'case_number': 1, 'override_spec_params': 'durability;replicas'} test_temp_MB (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-02-15 02:50:26,486 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= BaseTestCase setup started for test #1 test_temp_MB ========= 2021-02-15 02:50:27,374 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= BaseTestCase setup started for test #1 test_temp_MB ========= 2021-02-15 02:50:28,237 | test | INFO | MainThread | [basetestcase:setUp:311] Initializing cluster 2021-02-15 02:51:07,954 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:580] RAM quota was defined less than 100 MB: 2021-02-15 02:51:27,424 | test | INFO | MainThread | [basetestcase:setUp:333] Couchbase Cluster: default, Nodes: 172.23.105.215, 172.23.105.217, 172.23.105.219, 172.23.105.220, 172.23.106.237, 172.23.107.90 initialized 2021-02-15 02:51:27,446 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= BaseTestCase setup finished for test #1 test_temp_MB ========= 2021-02-15 02:51:27,470 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= CollectionBase setup started for test #1 test_temp_MB ========= 2021-02-15 02:51:35,240 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 1.78257594778 | Cluster node | | 172.23.105.217 | None | | | <--- IN --- | | 172.23.105.219 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 02:51:35,255 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 02:51:45,308 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-15 02:51:45,328 | test | INFO | pool-1-thread-7 | [task:check:379] Rebalance completed with progress: 100% in 10.0880000591 sec 2021-02-15 02:51:45,535 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1782] Creating required buckets from template 2021-02-15 02:51:51,046 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2238] Waiting for all collections to be created 2021-02-15 02:51:51,263 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | ephemeral | 2 | none | 0 | 0 | 314572800 | 41409296 | 0 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 314572800 | 40040816 | 0 | | default | ephemeral | 2 | none | 0 | 0 | 4718592000 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-02-15 02:51:51,265 | test | INFO | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-02-15 02:52:04,186 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 02:52:04,207 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 02:52:30,516 | test | INFO | pool-1-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *****...............**.......................................****..............................................*X | 1300.539 | | 967.539 | .........................................................................*X | 967.539 | +---------+-------------------------------------------------------------------------------------------------------------------+----------+ 2021-02-15 02:52:30,811 | test | INFO | pool-1-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *****..................................................................**..................................***X | 1003.300 | | 670.632 | .................................................................*X | 670.632 | | 504.132 | .................................................*X | 504.132 | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ 2021-02-15 02:52:31,552 | test | INFO | pool-1-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****....................*.....................**.................................****.........................**.**X | 19648.896 | +-------+----------------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 02:52:31,631 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.105.215 | kv | 90.3506526747 | 4201840640 | 3095142400 | 117440512 / 3758092288 | 115205 / 229635 | 7.0.0-4465-enterprise | | 172.23.105.217 | kv | 89.1662377766 | 4201844736 | 3061067776 | 6815744 / 3758092288 | 115563 / 228615 | 7.0.0-4465-enterprise | | 172.23.105.219 | kv | 87.3646209386 | 4201840640 | 3092508672 | 116916224 / 3758092288 | 143412 / 288721 | 7.0.0-4465-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-02-15 02:52:33,821 | test | INFO | pool-1-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:34,418 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:35,184 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:35,931 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:36,494 | test | INFO | pool-1-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:37,108 | test | INFO | pool-1-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:38,019 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:38,617 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:39,311 | test | INFO | pool-1-thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:39,930 | test | INFO | pool-1-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:40,526 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:41,381 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:42,127 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:42,664 | test | INFO | pool-1-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:43,226 | test | INFO | pool-1-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:44,186 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:44,677 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:45,536 | test | INFO | pool-1-thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:46,046 | test | INFO | pool-1-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:46,707 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:47,573 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:48,311 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:48,752 | test | INFO | pool-1-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:49,322 | test | INFO | pool-1-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:50,293 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:50,826 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:51,697 | test | INFO | pool-1-thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:52,180 | test | INFO | pool-1-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:52,956 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:53,861 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:54,377 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:54,940 | test | INFO | pool-1-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:55,448 | test | INFO | pool-1-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:56,448 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:56,967 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:57,903 | test | INFO | pool-1-thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:58,319 | test | INFO | pool-1-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:59,023 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:52:59,970 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:53:00,542 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:53:01,076 | test | INFO | pool-1-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:53:01,575 | test | INFO | pool-1-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:53:02,614 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:53:03,092 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:53:04,052 | test | INFO | pool-1-thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-15 02:53:09,059 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4388] Validating collection stats and item counts 2021-02-15 02:53:46,992 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | ephemeral | 2 | none | 0 | 20050 | 314572800 | 83629512 | 102 | | bucket2 | ephemeral | 2 | none | 0 | 15075 | 314572800 | 79292632 | 102 | | default | ephemeral | 2 | none | 0 | 500000 | 4718592000 | 751690552 | 102 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-02-15 02:53:46,993 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= CollectionBase setup complete for test #1 test_temp_MB ========= 2021-02-15 02:53:51,602 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 1} 2021-02-15 02:53:51,604 | test | INFO | MainThread | [collections_rebalance:test_temp_MB:851] Cycle 0 2021-02-15 02:53:51,651 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 02:53:51,657 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 02:53:59,789 | test | INFO | pool-1-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 19.2464358452 | Cluster node | | 172.23.105.217 | kv | 7.0.0-4465-enterprise | 15.0765306122 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4465-enterprise | 15.6782896411 | Cluster node | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 02:53:59,801 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 02:54:02,924 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-309 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-299 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:54:04,546 | test | ERROR | pool-1-thread-20 | [task:call:3557] Failed to load 2 docs from 0 to 1000 2021-02-15 02:54:04,898 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 0.438605527308 2021-02-15 02:54:05,026 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5135 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:54:05,029 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5135 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:54:05,247 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:54:05,249 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5234 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:54:06,582 | test | ERROR | pool-1-thread-20 | [task:call:3557] Failed to load 3 docs from 5000 to 5800 2021-02-15 02:54:06,918 | test | INFO | pool-1-thread-3 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:54:06,920 | test | INFO | pool-1-thread-3 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5774 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:54:08,069 | test | ERROR | pool-1-thread-20 | [task:call:3557] Failed to load 1 docs from 5000 to 5800 2021-02-15 02:54:09,967 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 2.44762636899 2021-02-15 02:54:14,234 | test | INFO | pool-1-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.........***.....................................................................*......................*X | 1022.900 | | 815.200 | ..............................................................................X | 815.200 | +---------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-02-15 02:54:15,006 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 5.50388252877 2021-02-15 02:54:15,727 | test | INFO | pool-1-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..............................**.....................................................................*.X | 816.900 | | 612.900 | ..........................................................................*X | 612.900 | | 179.034 | ....................X | 179.034 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 02:54:15,825 | test | INFO | pool-1-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **............**.......................................................................**.................*X | 20266.900 | | 14708.610 | .......................................................................X | 14708.610 | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 02:54:20,052 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 10.3805339778 2021-02-15 02:54:25,095 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 13.6736629066 2021-02-15 02:54:30,134 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 15.9628111745 2021-02-15 02:54:35,184 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 19.6298920956 2021-02-15 02:54:40,296 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 23.7447142328 2021-02-15 02:54:45,345 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 26.9929389482 2021-02-15 02:54:50,381 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 30.2222896675 2021-02-15 02:54:55,430 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 33.1376659679 2021-02-15 02:55:00,482 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 35.8648171447 2021-02-15 02:55:05,519 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 40.462216493 2021-02-15 02:55:10,581 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 45.2471372518 2021-02-15 02:55:15,661 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 48.9954935293 2021-02-15 02:55:20,704 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 52.3641184424 2021-02-15 02:55:25,756 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 55.2806560889 2021-02-15 02:55:30,812 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 58.4157512361 2021-02-15 02:55:35,855 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 61.569117761 2021-02-15 02:55:40,907 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 64.7236895226 2021-02-15 02:55:45,969 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 02:55:51,023 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 69.805790535 2021-02-15 02:55:56,073 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 73.636526311 2021-02-15 02:56:01,151 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 76.8439327382 2021-02-15 02:56:06,207 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 81.0108725143 2021-02-15 02:56:11,255 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 82.9892443811 2021-02-15 02:56:16,309 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 85.1240450953 2021-02-15 02:56:21,368 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 87.0493095894 2021-02-15 02:56:26,413 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 89.1004111601 2021-02-15 02:56:31,476 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 90.9727970981 2021-02-15 02:56:36,530 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 92.9185279404 2021-02-15 02:56:41,609 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 94.9591741773 2021-02-15 02:56:46,673 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 97.0171020581 2021-02-15 02:56:51,726 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: running, progress: 99.1771587589 2021-02-15 02:57:01,792 | test | INFO | pool-1-thread-10 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-15 02:57:01,823 | test | INFO | pool-1-thread-10 | [task:check:379] Rebalance completed with progress: 100% in 182.03399992 sec 2021-02-15 02:57:01,857 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 02:57:01,862 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 02:57:02,009 | test | INFO | pool-1-thread-16 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 63.7270341207 | Cluster node | | 172.23.106.237 | kv | 7.0.0-4465-enterprise | 54.8667897652 | --- OUT ---> | | 172.23.105.220 | kv | 7.0.0-4465-enterprise | 51.4106583072 | --- OUT ---> | | 172.23.105.217 | kv | 7.0.0-4465-enterprise | 51.3357778942 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4465-enterprise | 50.1180173092 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 02:57:02,026 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 02:57:05,466 | test | INFO | pool-1-thread-5 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2118 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2088 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:57:05,910 | test | INFO | pool-1-thread-5 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2867 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:57:06,338 | test | ERROR | pool-1-thread-1 | [task:call:3557] Failed to load 3 docs from 1960 to 2881 2021-02-15 02:57:07,096 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 0.479526070017 2021-02-15 02:57:08,220 | test | INFO | pool-1-thread-28 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2047 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:57:08,811 | test | INFO | pool-1-thread-20 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5974 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:57:08,812 | test | INFO | pool-1-thread-20 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5974 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:57:09,479 | test | ERROR | pool-1-thread-1 | [task:call:3557] Failed to load 1 docs from 1960 to 2881 2021-02-15 02:57:09,532 | test | ERROR | pool-1-thread-1 | [task:call:3557] Failed to load 1 docs from 5800 to 6568 2021-02-15 02:57:11,559 | test | INFO | pool-1-thread-26 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:57:11,561 | test | INFO | pool-1-thread-26 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6266 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 02:57:12,275 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 1.38381721694 2021-02-15 02:57:12,426 | test | ERROR | pool-1-thread-1 | [task:call:3557] Failed to load 1 docs from 5800 to 6568 2021-02-15 02:57:16,884 | test | INFO | pool-1-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.................................*......................*.............................................X | 1192.067 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-02-15 02:57:17,338 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 2.67943177761 2021-02-15 02:57:17,345 | test | INFO | pool-1-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.............................*.......................................................................*X | 542.493 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 02:57:17,365 | test | INFO | pool-1-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.......................*.............................................................................*X | 19172.039 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 02:57:22,381 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 4.93382848824 2021-02-15 02:57:27,415 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 7.24577335516 2021-02-15 02:57:32,463 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 9.52829871767 2021-02-15 02:57:37,509 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 11.9694795184 2021-02-15 02:57:42,555 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 15.6196215223 2021-02-15 02:57:47,598 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 19.2000756677 2021-02-15 02:57:52,637 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 23.1645721489 2021-02-15 02:57:57,686 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 27.0774786952 2021-02-15 02:58:02,732 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 31.0262656612 2021-02-15 02:58:07,773 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 02:58:12,819 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 02:58:18,411 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 33.6335824593 2021-02-15 02:58:23,447 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 36.8440506984 2021-02-15 02:58:28,497 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 39.2814646785 2021-02-15 02:58:33,551 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 41.4772572751 2021-02-15 02:58:38,585 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 43.7867785634 2021-02-15 02:58:43,648 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 46.2095474367 2021-02-15 02:58:48,687 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 49.1287315016 2021-02-15 02:58:53,724 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 52.5341306613 2021-02-15 02:58:58,766 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 56.3573711449 2021-02-15 02:59:03,815 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 60.1776719226 2021-02-15 02:59:08,865 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 64.0128826648 2021-02-15 02:59:13,911 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 02:59:18,956 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 02:59:24,624 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 02:59:29,667 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 68.2854224439 2021-02-15 02:59:34,711 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 69.6928119073 2021-02-15 02:59:39,753 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 71.0200933799 2021-02-15 02:59:44,790 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 72.6517543088 2021-02-15 02:59:49,835 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 73.9494988296 2021-02-15 02:59:54,872 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 75.5252841588 2021-02-15 02:59:59,924 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 76.9619519759 2021-02-15 03:00:04,959 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 78.7574497978 2021-02-15 03:00:09,992 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 80.9315708888 2021-02-15 03:00:15,046 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 82.975586826 2021-02-15 03:00:20,084 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 85.4927042318 2021-02-15 03:00:25,124 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 88.2448217883 2021-02-15 03:00:30,180 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 90.8091574199 2021-02-15 03:00:35,221 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 93.4183407637 2021-02-15 03:00:40,272 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 95.8901587123 2021-02-15 03:00:45,311 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 98.3212656132 2021-02-15 03:00:50,358 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 99.8879551821 2021-02-15 03:00:55,394 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-15 03:01:00,430 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-15 03:01:10,496 | test | INFO | pool-1-thread-16 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-15 03:01:10,591 | test | INFO | pool-1-thread-16 | [task:check:379] Rebalance completed with progress: 100% in 248.592999935 sec 2021-02-15 03:01:10,592 | test | INFO | MainThread | [collections_rebalance:test_temp_MB:851] Cycle 1 2021-02-15 03:01:10,628 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 03:01:10,637 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 03:01:18,903 | test | INFO | pool-1-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 18.7515923567 | Cluster node | | 172.23.105.217 | kv | 7.0.0-4465-enterprise | 14.7494887526 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4465-enterprise | 14.6646263708 | Cluster node | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 03:01:18,924 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:01:24,301 | test | INFO | pool-1-thread-6 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6595 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:01:24,302 | test | INFO | pool-1-thread-6 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6595 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:01:25,328 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:01:26,177 | test | ERROR | pool-1-thread-12 | [task:call:3557] Failed to load 1 docs from 6568 to 7305 2021-02-15 03:01:30,444 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 3.01734673291 2021-02-15 03:01:32,756 | test | INFO | pool-1-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.......***......................................................................*.......................**X | 1101.134 | +-------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-02-15 03:01:33,318 | test | INFO | pool-1-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.......................***...........................................................................*..**X | 481.176 | +-------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:01:34,372 | test | INFO | pool-1-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.....................***..........................................................................**.....*X | 19072.036 | | 13107.619 | ...................................................................X | 13107.619 | +-----------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 03:01:35,503 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 7.80086846901 2021-02-15 03:01:40,539 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 13.0465692239 2021-02-15 03:01:45,589 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 16.9483986615 2021-02-15 03:01:50,631 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 19.8822047309 2021-02-15 03:01:55,668 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 23.206812578 2021-02-15 03:02:00,707 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 26.5046965307 2021-02-15 03:02:05,743 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 29.7940196455 2021-02-15 03:02:10,802 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 33.1935709294 2021-02-15 03:02:15,842 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 35.6231898205 2021-02-15 03:02:20,898 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 40.2012512244 2021-02-15 03:02:25,954 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 45.585238909 2021-02-15 03:02:31,015 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 50.2172283082 2021-02-15 03:02:36,066 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 53.4964920834 2021-02-15 03:02:41,111 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 56.6139936116 2021-02-15 03:02:46,151 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 59.8359321324 2021-02-15 03:02:51,213 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 63.0406818033 2021-02-15 03:02:56,256 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 66.3032844165 2021-02-15 03:03:01,331 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 68.2926303129 2021-02-15 03:03:06,391 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 72.1789400644 2021-02-15 03:03:11,443 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 75.8056564066 2021-02-15 03:03:16,494 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 79.1143571301 2021-02-15 03:03:21,556 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 82.1995051066 2021-02-15 03:03:26,599 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 84.2318868274 2021-02-15 03:03:31,657 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 86.3245914816 2021-02-15 03:03:36,713 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 88.3349258779 2021-02-15 03:03:41,770 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 90.2815484384 2021-02-15 03:03:46,822 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 92.2199560941 2021-02-15 03:03:51,874 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 94.2393675304 2021-02-15 03:03:56,937 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 96.1567201128 2021-02-15 03:04:01,996 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 98.0453285133 2021-02-15 03:04:07,048 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: running, progress: 99.6366177498 2021-02-15 03:04:17,107 | test | INFO | pool-1-thread-15 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-15 03:04:17,144 | test | INFO | pool-1-thread-15 | [task:check:379] Rebalance completed with progress: 100% in 178.247999907 sec 2021-02-15 03:04:17,181 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 03:04:17,187 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 03:04:17,299 | test | INFO | pool-1-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 58.8807146611 | Cluster node | | 172.23.106.237 | kv | 7.0.0-4465-enterprise | 29.481865285 | --- OUT ---> | | 172.23.105.220 | kv | 7.0.0-4465-enterprise | 49.7256336556 | --- OUT ---> | | 172.23.105.217 | kv | 7.0.0-4465-enterprise | 27.9731127198 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4465-enterprise | 25.618556701 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 03:04:17,312 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:04:20,921 | test | INFO | pool-1-thread-4 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2943 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-3047 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:04:21,831 | test | ERROR | pool-1-thread-8 | [task:call:3557] Failed to load 2 docs from 2881 to 3765 2021-02-15 03:04:22,384 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 0.985502651569 2021-02-15 03:04:27,440 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 2.07436338529 2021-02-15 03:04:28,269 | test | INFO | pool-1-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3094 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-3104 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:04:28,414 | test | INFO | pool-1-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3676 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:04:29,020 | test | ERROR | pool-1-thread-8 | [task:call:3557] Failed to load 3 docs from 2881 to 3765 2021-02-15 03:04:32,486 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 3.52018448474 2021-02-15 03:04:33,542 | test | INFO | pool-1-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **......................................*.........................*.....................................*X | 533.350 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:04:34,029 | test | INFO | pool-1-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.......................................*........................*.....................................*X | 525.960 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:04:35,874 | test | INFO | pool-1-thread-27 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.................*..............*.....................*.................................*...............X | 22605.371 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 03:04:37,532 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 5.97641246493 2021-02-15 03:04:42,602 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 8.14365402655 2021-02-15 03:04:47,634 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 10.4248941951 2021-02-15 03:04:52,704 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 12.7908896373 2021-02-15 03:04:57,746 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 15.6914708938 2021-02-15 03:05:02,790 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 19.0591539678 2021-02-15 03:05:07,835 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 22.9739205589 2021-02-15 03:05:12,875 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 26.7959876374 2021-02-15 03:05:17,921 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 30.7542739517 2021-02-15 03:05:22,961 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 03:05:28,012 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 03:05:33,615 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 33.4834578963 2021-02-15 03:05:38,657 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 37.0209865656 2021-02-15 03:05:43,688 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 39.1964101278 2021-02-15 03:05:48,743 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 41.5055383949 2021-02-15 03:05:53,775 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 43.7019352042 2021-02-15 03:05:58,816 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 46.0122075146 2021-02-15 03:06:03,852 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 48.9035299762 2021-02-15 03:06:08,904 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 52.3477570362 2021-02-15 03:06:13,940 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 56.2173709794 2021-02-15 03:06:18,980 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 60.1882480651 2021-02-15 03:06:24,016 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 64.2379961046 2021-02-15 03:06:29,059 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 03:06:34,142 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 03:06:39,473 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 66.7885101099 2021-02-15 03:06:44,516 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 68.5529166962 2021-02-15 03:06:49,559 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 70.6436936053 2021-02-15 03:06:54,592 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 71.9934118064 2021-02-15 03:06:59,631 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 73.2058024994 2021-02-15 03:07:04,677 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 74.6687332649 2021-02-15 03:07:09,714 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 75.9944302665 2021-02-15 03:07:14,750 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 77.4017259857 2021-02-15 03:07:19,802 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 78.6988719523 2021-02-15 03:07:24,848 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 80.440389165 2021-02-15 03:07:29,888 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 82.7929453005 2021-02-15 03:07:34,931 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 84.5391091204 2021-02-15 03:07:39,970 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 86.7409392721 2021-02-15 03:07:45,007 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 89.3628873674 2021-02-15 03:07:50,049 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 91.7853684472 2021-02-15 03:07:55,086 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 94.6747870167 2021-02-15 03:08:00,125 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 97.0172690279 2021-02-15 03:08:05,173 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 98.8803152288 2021-02-15 03:08:10,213 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-15 03:08:15,255 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-15 03:08:20,427 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-15 03:08:30,483 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-15 03:08:30,585 | test | INFO | pool-1-thread-5 | [task:check:379] Rebalance completed with progress: 100% in 253.286000013 sec 2021-02-15 03:08:30,585 | test | INFO | MainThread | [collections_rebalance:test_temp_MB:851] Cycle 2 2021-02-15 03:08:30,621 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 03:08:30,628 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 03:08:33,730 | test | INFO | pool-1-thread-14 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5135 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:33,762 | test | INFO | pool-1-thread-14 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:34,632 | test | ERROR | pool-1-thread-9 | [task:call:3557] Failed to load 3 docs from 4614 to 5429 2021-02-15 03:08:38,930 | test | INFO | pool-1-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 18.6687069625 | Cluster node | | 172.23.105.217 | kv | 7.0.0-4465-enterprise | 14.3914263843 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4465-enterprise | 14.4605967865 | Cluster node | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 03:08:38,970 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:08:45,480 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8151 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-8084 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-8114 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:45,480 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8151 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8084 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8114 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:45,640 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-8338 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:45,641 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8274 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8338 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:45,789 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8496 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-8506 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:45,790 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8496 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8506 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:45,842 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:08:46,151 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8623 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-8666 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:46,151 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8623 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8666 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:08:47,292 | test | ERROR | pool-1-thread-9 | [task:call:3557] Failed to load 9 docs from 8013 to 8692 2021-02-15 03:08:50,404 | test | INFO | pool-1-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..................***..........................................................................*........*X | 893.973 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:08:50,933 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 3.70538395414 2021-02-15 03:08:51,835 | test | INFO | pool-1-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **......................**.......................................*.......................................X | 694.730 | | 640.544 | ...........................................................................................*X | 640.544 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:08:51,851 | test | INFO | pool-1-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...................**...........................................................................**......*X | 18852.346 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 03:08:55,986 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 9.1333215946 2021-02-15 03:09:01,038 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 13.8178596747 2021-02-15 03:09:06,082 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 17.5624678683 2021-02-15 03:09:11,128 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 20.6739837797 2021-02-15 03:09:16,198 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 23.8538674849 2021-02-15 03:09:21,244 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 27.0720629633 2021-02-15 03:09:26,302 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 30.4126094868 2021-02-15 03:09:31,357 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 03:09:36,392 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 36.5834356013 2021-02-15 03:09:41,463 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 41.6358681904 2021-02-15 03:09:46,595 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 46.8362106895 2021-02-15 03:09:51,647 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 50.8682766877 2021-02-15 03:09:56,703 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 54.2208075788 2021-02-15 03:10:01,756 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 57.3296244126 2021-02-15 03:10:06,813 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 60.4921374382 2021-02-15 03:10:11,869 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 63.7768349319 2021-02-15 03:10:16,937 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 03:10:22,000 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 69.2738465491 2021-02-15 03:10:27,039 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 72.3018044449 2021-02-15 03:10:32,088 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 75.7319318871 2021-02-15 03:10:37,150 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 79.2090019813 2021-02-15 03:10:42,211 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 81.7998557497 2021-02-15 03:10:47,273 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 84.04284922 2021-02-15 03:10:52,326 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 86.3148250848 2021-02-15 03:10:57,368 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 87.940066996 2021-02-15 03:11:02,424 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 89.8146934506 2021-02-15 03:11:07,467 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 91.7053543672 2021-02-15 03:11:12,515 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 93.6021656796 2021-02-15 03:11:17,572 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 95.3632830097 2021-02-15 03:11:22,614 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 97.184579649 2021-02-15 03:11:27,661 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: running, progress: 98.9006001132 2021-02-15 03:11:37,727 | test | INFO | pool-1-thread-24 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-15 03:11:37,765 | test | INFO | pool-1-thread-24 | [task:check:379] Rebalance completed with progress: 100% in 178.840000153 sec 2021-02-15 03:11:37,802 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 03:11:37,808 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 03:11:37,921 | test | INFO | pool-1-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 50.091503268 | Cluster node | | 172.23.106.237 | kv | 7.0.0-4465-enterprise | 44.9059561129 | --- OUT ---> | | 172.23.105.220 | kv | 7.0.0-4465-enterprise | 48.5393844549 | --- OUT ---> | | 172.23.105.217 | kv | 7.0.0-4465-enterprise | 43.7007874016 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4465-enterprise | 44.7843137255 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 03:11:37,936 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:11:39,329 | test | INFO | pool-1-thread-22 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:39,782 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 1 docs from 5429 to 6212 2021-02-15 03:11:40,502 | test | INFO | pool-1-thread-10 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5974 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:40,650 | test | INFO | pool-1-thread-7 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5890 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5900 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:40,875 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 2 docs from 5429 to 6212 2021-02-15 03:11:40,937 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 1 docs from 5429 to 6212 2021-02-15 03:11:41,654 | test | INFO | pool-1-thread-28 | [table_view:display:72] Keys failed in bucket1:_default:_default +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-22 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:42,671 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 1 docs from 21 to 25 2021-02-15 03:11:42,986 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 0.535158324452 2021-02-15 03:11:48,058 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 1.60547497336 2021-02-15 03:11:48,167 | test | INFO | pool-1-thread-24 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:48,167 | test | INFO | pool-1-thread-24 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9275 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:48,174 | test | INFO | pool-1-thread-28 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4673 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:48,265 | test | INFO | pool-1-thread-28 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4872 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:48,342 | test | INFO | pool-1-thread-28 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5176 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:48,437 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 1 docs from 8692 to 9344 2021-02-15 03:11:49,131 | test | INFO | pool-1-thread-12 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4837 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:49,548 | test | INFO | pool-1-thread-12 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5133 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:49,677 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 3 docs from 4614 to 5429 2021-02-15 03:11:49,760 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9326 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:49,762 | test | INFO | pool-1-thread-23 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9326 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:11:49,792 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 2 docs from 4614 to 5429 2021-02-15 03:11:49,801 | test | ERROR | pool-1-thread-11 | [task:call:3557] Failed to load 1 docs from 8692 to 9344 2021-02-15 03:11:51,502 | test | INFO | pool-1-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..................................*..............................*....................................X | 403.715 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:11:52,681 | test | INFO | pool-1-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **........................**...........................................................................*X | 390.112 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:11:53,134 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 4.11657815102 2021-02-15 03:11:53,490 | test | INFO | pool-1-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **......................................*............................*..................................*X | 14502.702 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 03:11:58,187 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 6.2867006948 2021-02-15 03:12:03,226 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 8.51098560268 2021-02-15 03:12:08,273 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 10.8763768321 2021-02-15 03:12:13,312 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 12.9033274765 2021-02-15 03:12:18,358 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 15.8480376518 2021-02-15 03:12:23,401 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 19.1984031111 2021-02-15 03:12:28,443 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 23.3465209997 2021-02-15 03:12:33,483 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 27.3551085125 2021-02-15 03:12:38,532 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 31.2798126886 2021-02-15 03:12:43,582 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 03:12:48,631 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 03:12:53,954 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 33.7086300599 2021-02-15 03:12:59,032 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 36.9371025533 2021-02-15 03:13:04,068 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 39.3379681943 2021-02-15 03:13:09,112 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 41.5334908756 2021-02-15 03:13:14,151 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 43.7019352042 2021-02-15 03:13:19,211 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 46.1815362322 2021-02-15 03:13:24,253 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 48.7643101947 2021-02-15 03:13:29,289 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 52.1145875685 2021-02-15 03:13:34,336 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 55.8448936446 2021-02-15 03:13:39,372 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 59.9442619015 2021-02-15 03:13:44,414 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 63.7591301044 2021-02-15 03:13:49,450 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 03:13:54,496 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 03:13:59,598 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 03:14:04,701 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 68.0372030797 2021-02-15 03:14:09,743 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 69.9355249615 2021-02-15 03:14:14,801 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 71.4008228479 2021-02-15 03:14:19,834 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 72.7275938518 2021-02-15 03:14:24,888 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 74.0765960513 2021-02-15 03:14:29,924 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 75.4037837798 2021-02-15 03:14:34,954 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 76.7247747748 2021-02-15 03:14:40,009 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 78.023141553 2021-02-15 03:14:45,039 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 79.3167198977 2021-02-15 03:14:50,075 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 80.7707556026 2021-02-15 03:14:55,121 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 82.9918893116 2021-02-15 03:15:00,154 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 85.0722046589 2021-02-15 03:15:05,203 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 87.2547719323 2021-02-15 03:15:10,259 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 89.7347130829 2021-02-15 03:15:15,293 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 92.1576108872 2021-02-15 03:15:20,332 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 94.9538544958 2021-02-15 03:15:25,371 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 97.2138314328 2021-02-15 03:15:30,415 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 98.9362201904 2021-02-15 03:15:35,453 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-15 03:15:40,493 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-15 03:15:50,546 | test | INFO | pool-1-thread-4 | [task:check:322] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.105.220:8091 during connect (<_realsocket at 0x68ad4 type=client open_count=1 channel=[id: 0x4f0c9f5a, 0.0.0.0/0.0.0.0:52749] timeout=120.0>) 2021-02-15 03:15:50,582 | test | ERROR | pool-1-thread-4 | [rest_client:_http_request:751] Socket error while connecting to http://172.23.105.220:8091/nodes/self error [Errno 111] Connection refused 2021-02-15 03:15:52,658 | test | INFO | pool-1-thread-4 | [task:check:379] Rebalance completed with progress: 100% in 254.75 sec 2021-02-15 03:15:52,660 | test | INFO | MainThread | [collections_rebalance:test_temp_MB:851] Cycle 3 2021-02-15 03:15:52,710 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-15 03:15:52,716 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-15 03:15:53,078 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:15:53,161 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in bucket1:scope1:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6595 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:15:53,278 | test | ERROR | pool-1-thread-10 | [task:call:3557] Failed to load 2 docs from 6212 to 6963 2021-02-15 03:15:59,921 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4465-enterprise | 16.1618732502 | Cluster node | | 172.23.105.217 | kv | 7.0.0-4465-enterprise | 13.183444047 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4465-enterprise | 11.3653699466 | Cluster node | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-15 03:15:59,946 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:16:05,697 | test | INFO | pool-1-thread-17 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9835 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:16:05,697 | test | INFO | pool-1-thread-17 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9835 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-15 03:16:06,392 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-15 03:16:06,602 | test | ERROR | pool-1-thread-10 | [task:call:3557] Failed to load 1 docs from 9344 to 9970 2021-02-15 03:16:11,460 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 3.36379314738 2021-02-15 03:16:11,584 | test | INFO | pool-1-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.........***.........................................................................**..................*X | 957.500 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:16:12,398 | test | INFO | pool-1-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *......***.......................................................................**.......................*X | 738.800 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-02-15 03:16:12,651 | test | INFO | pool-1-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........**.....................................................................*.......................**X | 17029.200 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-15 03:16:16,502 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 8.17655535008 2021-02-15 03:16:21,540 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 13.3381286725 2021-02-15 03:16:26,602 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 17.8433960679 2021-02-15 03:16:31,658 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 21.09850347 2021-02-15 03:16:36,700 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 24.2594263659 2021-02-15 03:16:41,750 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 27.4968525688 2021-02-15 03:16:46,782 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 30.7394622678 2021-02-15 03:16:51,821 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-15 03:16:56,871 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 37.033116992 2021-02-15 03:17:01,923 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 42.388439061 2021-02-15 03:17:07,028 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 47.6476199681 2021-02-15 03:17:12,085 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 51.702551412 2021-02-15 03:17:17,135 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 54.6033856828 2021-02-15 03:17:22,180 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 57.7766510219 2021-02-15 03:17:27,226 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 61.0928469565 2021-02-15 03:17:32,279 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 64.3781107759 2021-02-15 03:17:37,322 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-15 03:17:42,367 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 69.674845557 2021-02-15 03:17:47,414 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 72.5996830574 2021-02-15 03:17:52,464 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 75.6464843152 2021-02-15 03:17:57,516 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 78.7311512597 2021-02-15 03:18:02,578 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 80.6884058197 2021-02-15 03:18:07,617 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 83.4335146311 2021-02-15 03:18:12,663 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 85.4108975973 2021-02-15 03:18:17,710 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 87.482746302 2021-02-15 03:18:22,759 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 89.2443334965 2021-02-15 03:18:27,806 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 91.0542556396 2021-02-15 03:18:32,852 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 92.9309971199 2021-02-15 03:18:37,897 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 94.7859732587 2021-02-15 03:18:42,947 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:18:48,000 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:18:53,055 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:18:58,095 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:03,151 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:08,206 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:13,259 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:18,313 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:23,362 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:28,417 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:33,470 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:38,523 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:43,579 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:48,630 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:53,683 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:19:58,736 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:03,789 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:08,836 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:13,894 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:18,947 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:24,002 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:29,045 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:34,095 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:39,151 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 96.4371664552 2021-02-15 03:20:44,200 | test | ERROR | pool-1-thread-7 | [rest_client:_rebalance_status_and_progress:1485] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-02-15 03:20:44,255 | test | INFO | pool-1-thread-7 | [rest_client:print_UI_logs:2580] Latest logs from UI on 172.23.105.215: 2021-02-15 03:20:44,256 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.237', u'tstamp': 1613388043218L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:20:43.218Z', u'text': u"Control connection to memcached on 'ns_1@172.23.106.237' disconnected. Check logs for details."} 2021-02-15 03:20:44,256 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613388042485L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:20:42.485Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.22641.49>,\n {{wait_seqno_persisted_failed,"default",242,\n 7708,\n [{\'ns_1@172.23.106.237\',\n {\'EXIT\',\n {{{badmatch,{error,timeout}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,158}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,350}]},\n {ns_memcached_sockets_pool,\n take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,46}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,88}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},\n {line,197}]}]},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.106.237\'},\n {if_rebalance,<0.23245.47>,\n {wait_seqno_persisted,242,7708}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,488}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 3c59b4de583c532410f1cc4384aa10aa'} 2021-02-15 03:20:44,256 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613388042452L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:20:42.452Z', u'text': u'Worker <0.22558.49> (for action {move,{242,\n [\'ns_1@172.23.105.217\',\n \'ns_1@172.23.105.215\',\n \'ns_1@172.23.105.219\'],\n [\'ns_1@172.23.105.220\',\n \'ns_1@172.23.106.237\',\n \'ns_1@172.23.105.217\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.22641.49>,\n {{wait_seqno_persisted_failed,\n "default",\n 242,7708,\n [{\'ns_1@172.23.106.237\',\n {\'EXIT\',\n {{{badmatch,\n {error,\n timeout}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 158}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 350}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 46}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 88}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 197}]}]},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.106.237\'},\n {if_rebalance,\n <0.23245.47>,\n {wait_seqno_persisted,\n 242,\n 7708}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 488}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2021-02-15 03:20:44,257 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613387857585L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:17:37.585Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2021-02-15 03:20:44,257 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.237', u'tstamp': 1613387855976L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:17:35.976Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.237\' in 0 seconds.'} 2021-02-15 03:20:44,259 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.220', u'tstamp': 1613387855975L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:17:35.975Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.105.220\' in 0 seconds.'} 2021-02-15 03:20:44,259 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613387855879L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:17:35.879Z', u'text': u'Started rebalancing bucket default'} 2021-02-15 03:20:44,259 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613387812572L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:16:52.572Z', u'text': u'Bucket "bucket2" rebalance does not seem to be swap rebalance'} 2021-02-15 03:20:44,259 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.237', u'tstamp': 1613387811021L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:16:51.021Z', u'text': u'Bucket "bucket2" loaded on node \'ns_1@172.23.106.237\' in 0 seconds.'} 2021-02-15 03:20:44,259 | test | ERROR | pool-1-thread-7 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.220', u'tstamp': 1613387811017L, u'shortText': u'message', u'serverTime': u'2021-02-15T03:16:51.017Z', u'text': u'Bucket "bucket2" loaded on node \'ns_1@172.23.105.220\' in 0 seconds.'} 2021-02-15 03:20:44,260 | test | ERROR | pool-1-thread-7 | [task:call:237] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed FAIL 2021-02-15 03:20:44,470 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ | bucket1 | ephemeral | 2 | none | 0 | 12011 | 524288000 | 88265928 | 170 | | bucket2 | ephemeral | 2 | none | 0 | 9030 | 524288000 | 79943024 | 170 | | default | ephemeral | 2 | none | 0 | 305200 | 7864320000 | 1027445152 | 170 | +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ 2021-02-15 03:20:45,493 | test | INFO | MainThread | [collections_base:tearDown:90] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-02-15 03:20:45,494 | test | INFO | MainThread | [collections_base:tearDown:93] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-02-15 03:20:46,183 | test | INFO | MainThread | [collections_base:tearDown:90] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-02-15 03:20:46,183 | test | INFO | MainThread | [collections_base:tearDown:93] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-02-15 03:20:46,868 | test | INFO | MainThread | [collections_base:tearDown:90] Bucket: default, Active Resident ratio(DGM): 100% 2021-02-15 03:20:46,868 | test | INFO | MainThread | [collections_base:tearDown:93] Bucket: default, Replica Resident ratio(DGM): 100% [ERROR] (cb-io-kv-89-3) Operator called default onErrorDropped - com.couchbase.client.core.error.RequestCanceledException: CarrierGlobalConfigRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) {"cancelled":true,"completed":true,"coreId":"0x5c6352ae00000016","idempotent":true,"lastChannelId":"5C6352AE00000016/000000001857A4B3","lastDispatchedFrom":"172.23.123.70:55904","lastDispatchedTo":"172.23.106.237:11210","reason":"NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)","requestId":3246151,"requestType":"CarrierGlobalConfigRequest","retried":0,"service":{"opaque":"0x33bade","target":"172.23.106.237","type":"kv"},"timeoutMs":2500} com.couchbase.client.core.error.RequestCanceledException: CarrierGlobalConfigRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) {"cancelled":true,"completed":true,"coreId":"0x5c6352ae00000016","idempotent":true,"lastChannelId":"5C6352AE00000016/000000001857A4B3","lastDispatchedFrom":"172.23.123.70:55904","lastDispatchedTo":"172.23.106.237:11210","reason":"NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT)","requestId":3246151,"requestType":"CarrierGlobalConfigRequest","retried":0,"service":{"opaque":"0x33bade","target":"172.23.106.237","type":"kv"},"timeoutMs":2500} at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:165) at com.couchbase.client.core.retry.RetryOrchestrator.lambda$maybeRetry$0(RetryOrchestrator.java:79) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792) at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153) at com.couchbase.client.core.retry.RetryOrchestrator.maybeRetry(RetryOrchestrator.java:62) at com.couchbase.client.core.io.netty.kv.KeyValueMessageHandler.channelInactive(KeyValueMessageHandler.java:246) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818) at com.couchbase.client.core.deps.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at com.couchbase.client.core.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2021-02-15 03:22:59,551 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:419] 172.23.105.215 node pcap zip copied on client : True 2021-02-15 03:24:28,760 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:419] 172.23.105.217 node pcap zip copied on client : True 2021-02-15 03:25:59,342 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:419] 172.23.105.219 node pcap zip copied on client : True 2021-02-15 03:26:52,378 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:419] 172.23.105.220 node pcap zip copied on client : True 2021-02-15 03:27:43,917 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:419] 172.23.106.237 node pcap zip copied on client : True 2021-02-15 03:27:45,789 | test | INFO | MainThread | [basetestcase:start_fetch_pcaps:419] 172.23.107.90 node pcap zip copied on client : True 2021-02-15 03:28:00,915 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1079] Running cbcollect on node ns_1@172.23.105.215,ns_1@172.23.105.217,ns_1@172.23.105.219,ns_1@172.23.105.220,ns_1@172.23.106.237 2021-02-15 03:28:10,941 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1082] ns_1@172.23.105.215,ns_1@172.23.105.217,ns_1@172.23.105.219,ns_1@172.23.105.220,ns_1@172.23.106.237 - cbcollect status: True 2021-02-15 03:28:10,943 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1087] Polling active_tasks to check cbcollect status 2021-02-15 03:38:12,164 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.105.215: Copying cbcollect ZIP file to Client 2021-02-15 03:38:13,743 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.105.215 node cb collect zip copied on client : True 2021-02-15 03:38:13,743 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.105.217: Copying cbcollect ZIP file to Client 2021-02-15 03:38:14,973 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.105.217 node cb collect zip copied on client : True 2021-02-15 03:38:14,973 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.105.219: Copying cbcollect ZIP file to Client 2021-02-15 03:38:16,193 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.105.219 node cb collect zip copied on client : True 2021-02-15 03:38:16,194 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.105.220: Copying cbcollect ZIP file to Client 2021-02-15 03:38:17,401 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1131] 172.23.105.220 node cb collect zip copied on client : True 2021-02-15 03:38:17,403 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1110] ns_1@172.23.106.237: Copying cbcollect ZIP file to Client cp: cannot stat ‘/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-02-15T112800-ns_1@172.23.106.237.zip’: No such file or directory Traceback (most recent call last): File "pytests/basetestcase.py", line 464, in tearDownEverything self.fetch_cb_collect_logs() File "pytests/basetestcase.py", line 607, in fetch_cb_collect_logs cluster_util.copy_cb_collect_logs(rest, nodes, cluster, File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 1125, in copy_cb_collect_logs cb_collect_size = int(os.path.getsize( File "/opt/jython/Lib/genericpath.py", line 49, in getsize 2021-02-15 03:38:18,315 | test | WARNING | MainThread | [basetestcase:tearDownEverything:491] Killing memcached due to [Errno 2] No such file or directory: u'/data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-15_02-49-48/test_1/collectinfo-2021-02-15T112800-ns_1@172.23.106.237.zip' return os.stat(filename).st_size OSError: [Errno 2] No such file or directory: u'/data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-15_02-49-48/test_1/collectinfo-2021-02-15T112800-ns_1@172.23.106.237.zip' ====================================================================== FAIL: test_temp_MB (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 858, in test_temp_MB self.wait_for_rebalance_to_complete(operation) File "pytests/bucket_collections/collections_rebalance.py", line 707, in wait_for_rebalance_to_complete self.assertTrue(task.result, "Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 2913.032s FAILED (failures=1) During the test, Remote Connections: 115, Disconnections: 115 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 1 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB testrunner logs, diags and results are available under /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-15_02-49-48/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 48m 42s 2 actionable tasks: 2 executed Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 7.0.0-4465,product=cb,parallel=True' [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE