Started by user Sumedh Basarkod [EnvInject] - Loading node environment variables. Building remotely on slv-s63702-24g-12c (P0 jython_slave IPv6 bkrs3) in workspace /data/workspace/rebalance_sumedh_temp [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/rebalance_sumedh_temp # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 5cd10ac5ee2186350ccf68b2f91baa67e626dec6 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 5cd10ac5ee2186350ccf68b2f91baa67e626dec6 > /usr/bin/git rev-list 5cd10ac5ee2186350ccf68b2f91baa67e626dec6 # timeout=10 [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins3866231992179773855.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:172.23.105.215 port:8091 [_2] ip:172.23.105.217 port:8091 [_3] ip:172.23.105.219 port:8091 [_4] ip:172.23.105.220 port:8091 [_5] ip:172.23.106.237 port:8091 [_6] ip:172.23.107.90 port:8091' + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + git fetch http://review.couchbase.org/TAF refs/changes/88/145788/4 From http://review.couchbase.org/TAF * branch refs/changes/88/145788/4 -> FETCH_HEAD + git cherry-pick FETCH_HEAD [detached HEAD 5022f93] Debug to repro MB Author: sumedhpb 1 file changed, 28 insertions(+), 1 deletion(-) [rebalance_sumedh_temp] $ /bin/sh -xe /tmp/jenkins4757306932931099885.sh + guides/gradlew --refresh-dependencies install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p version=7.0.0-4437,product=cb,parallel=True' Starting a Gradle Daemon, 30 busy Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p version=7.0.0-4437,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p version=7.0.0-4437,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p version=7.0.0-4437,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p version=7.0.0-4437,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p version=7.0.0-4437,product=cb,parallel=True > Task :install 2021-02-11 05:26:58,441 - root - INFO - Done with uninstall on 172.23.105.220. 2021-02-11 05:26:58,493 - root - INFO - Done with uninstall on 172.23.106.237. 2021-02-11 05:26:59,052 - root - INFO - Done with uninstall on 172.23.107.90. 2021-02-11 05:26:59,375 - root - INFO - Done with uninstall on 172.23.105.217. 2021-02-11 05:26:59,911 - root - INFO - Done with uninstall on 172.23.105.215. 2021-02-11 05:28:08,444 - root - INFO - Done with uninstall on 172.23.105.219. 2021-02-11 05:31:37,816 - root - INFO - Done with install on 172.23.107.90. 2021-02-11 05:31:38,632 - root - INFO - Done with install on 172.23.105.220. 2021-02-11 05:31:40,148 - root - INFO - Done with install on 172.23.105.215. 2021-02-11 05:31:41,615 - root - INFO - Done with install on 172.23.106.237. 2021-02-11 05:31:43,302 - root - INFO - Done with install on 172.23.105.217. 2021-02-11 05:32:39,532 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.107.90 2021-02-11 05:32:39,693 - root - INFO - Done with init on 172.23.107.90. 2021-02-11 05:32:39,888 - root - INFO - Done with cleanup on 172.23.107.90. 2021-02-11 05:32:40,009 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.220 2021-02-11 05:32:40,161 - root - INFO - Done with init on 172.23.105.220. 2021-02-11 05:32:40,384 - root - INFO - Done with cleanup on 172.23.105.220. 2021-02-11 05:32:41,520 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.215 2021-02-11 05:32:41,686 - root - INFO - Done with init on 172.23.105.215. 2021-02-11 05:32:41,882 - root - INFO - Done with cleanup on 172.23.105.215. 2021-02-11 05:32:42,974 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.237 2021-02-11 05:32:43,155 - root - INFO - Done with init on 172.23.106.237. 2021-02-11 05:32:43,410 - root - INFO - Done with cleanup on 172.23.106.237. 2021-02-11 05:32:44,670 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.217 2021-02-11 05:32:44,858 - root - INFO - Done with init on 172.23.105.217. 2021-02-11 05:32:45,061 - root - INFO - Done with cleanup on 172.23.105.217. 2021-02-11 05:32:53,890 - root - INFO - Done with install on 172.23.105.219. 2021-02-11 05:33:55,266 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.219 2021-02-11 05:33:55,423 - root - INFO - Done with init on 172.23.105.219. 2021-02-11 05:33:55,598 - root - INFO - Done with cleanup on 172.23.105.219. 2021-02-11 05:34:03,497 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-11 05:34:03,516 - root - INFO - node:172.23.105.215:8091 version:7.0.0-4437-enterprise aFamily:inet services:[u'kv'] 2021-02-11 05:34:03,536 - root - INFO - node:172.23.105.217:8091 version:7.0.0-4437-enterprise aFamily:inet services:[u'kv'] 2021-02-11 05:34:03,555 - root - INFO - node:172.23.105.219:8091 version:7.0.0-4437-enterprise aFamily:inet services:[u'kv'] 2021-02-11 05:34:03,573 - root - INFO - node:172.23.105.220:8091 version:7.0.0-4437-enterprise aFamily:inet services:[u'kv'] 2021-02-11 05:34:03,592 - root - INFO - node:172.23.106.237:8091 version:7.0.0-4437-enterprise aFamily:inet services:[u'kv'] 2021-02-11 05:34:03,611 - root - INFO - node:172.23.107.90:8091 version:7.0.0-4437-enterprise aFamily:inet services:[u'kv'] 2021-02-11 05:34:03,612 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-11 05:34:03,612 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-11 05:34:03,614 - root - INFO - INSTALL COMPLETED ON: 172.23.105.215 2021-02-11 05:34:03,614 - root - INFO - INSTALL COMPLETED ON: 172.23.105.217 2021-02-11 05:34:03,614 - root - INFO - INSTALL COMPLETED ON: 172.23.105.219 2021-02-11 05:34:03,615 - root - INFO - INSTALL COMPLETED ON: 172.23.105.220 2021-02-11 05:34:03,615 - root - INFO - INSTALL COMPLETED ON: 172.23.106.237 2021-02-11 05:34:03,615 - root - INFO - INSTALL COMPLETED ON: 172.23.107.90 2021-02-11 05:34:03,615 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-02-11 05:34:03,619 - root - INFO - TOTAL INSTALL TIME = 471.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 8m 4s 1 actionable task: 1 executed + guides/gradlew testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,data_load_stage=during,skip_validations=False -m rest' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,data_load_stage=during,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,data_load_stage=during,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,data_load_stage=during,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,data_load_stage=during,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/rebalance_sumedh_temp/build/classes/java/main:/data/workspace/rebalance_sumedh_temp/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p rerun=False,get-cbcollect-info=False,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,data_load_stage=during,skip_validations=False -m rest > Task :compileJava Note: /data/workspace/rebalance_sumedh_temp/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Global Test input params: {'cluster_name': 'durability_volume', 'collect_pcaps': 'True', 'get-cbcollect-info': 'False', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 6, 'rerun': 'False'} Logs will be stored at /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-11_05-34-11/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini rerun=False,get-cbcollect-info=False,collect_pcaps=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,data_load_stage=during,skip_validations=False' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'durability_volume', 'ini': '/tmp/durability_volume.ini', 'get-cbcollect-info': 'False', 'replicas': '2', 'durability': 'MAJORITY', 'nodes_in': '2', 'skip_validations': 'False', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-11_05-34-11/test_1', 'nodes_init': '3', 'collect_pcaps': 'True', 'bucket_spec': 'multi_bucket.buckets_all_ephemeral', 'case_number': 1, 'override_spec_params': 'durability;replicas'} test_temp_MB (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-02-11 05:34:34,318 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= BaseTestCase setup started for test #1 test_temp_MB ========= 2021-02-11 05:34:35,306 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= BaseTestCase setup started for test #1 test_temp_MB ========= 2021-02-11 05:34:36,372 | test | INFO | MainThread | [basetestcase:setUp:311] Initializing cluster 2021-02-11 05:35:15,956 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:580] RAM quota was defined less than 100 MB: 2021-02-11 05:35:35,244 | test | INFO | MainThread | [basetestcase:setUp:333] Couchbase Cluster: default, Nodes: 172.23.105.215, 172.23.105.217, 172.23.105.219, 172.23.105.220, 172.23.106.237, 172.23.107.90 initialized 2021-02-11 05:35:35,266 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= BaseTestCase setup finished for test #1 test_temp_MB ========= 2021-02-11 05:35:35,339 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= CollectionBase setup started for test #1 test_temp_MB ========= 2021-02-11 05:35:42,851 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4437-enterprise | 1.64099974754 | Cluster node | | 172.23.105.217 | None | | | <--- IN --- | | 172.23.105.219 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-11 05:35:42,868 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-11 05:35:52,928 | test | INFO | pool-1-thread-7 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-11 05:35:52,950 | test | INFO | pool-1-thread-7 | [task:check:379] Rebalance completed with progress: 100% in 10.0999999046 sec 2021-02-11 05:35:53,181 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1782] Creating required buckets from template 2021-02-11 05:35:58,947 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2238] Waiting for all collections to be created 2021-02-11 05:35:59,184 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | ephemeral | 2 | none | 0 | 0 | 314572800 | 0 | 0 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 314572800 | 18805144 | 34 | | default | ephemeral | 2 | none | 0 | 0 | 4718592000 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-02-11 05:35:59,184 | test | INFO | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-02-11 05:36:13,828 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-11 05:36:13,846 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-11 05:36:38,782 | test | INFO | pool-1-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****..........................................**.................................****.........................*X | 21302.696 | | 20750.279 | ................................................................................................X | 20750.279 | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-11 05:36:39,752 | test | INFO | pool-1-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******.............***.................................................**.....................................*X | 676.380 | +-------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-02-11 05:36:40,852 | test | INFO | pool-1-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..****.....................................................................**.............................**X | 975.066 | | 640.166 | ................................................................X | 640.166 | | 304.766 | ..............................*X | 304.766 | | 209.300 | ....................*X | 209.300 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-02-11 05:36:40,930 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ | 172.23.105.215 | kv | 89.0146642655 | 4201840640 | 3065933824 | 117440512 / 3758092288 | 120823 / 240049 | 7.0.0-4437-enterprise | | 172.23.105.217 | kv | 87.3319544984 | 4201844736 | 3063603200 | 6815744 / 3758092288 | 120412 / 239558 | 7.0.0-4437-enterprise | | 172.23.105.219 | kv | 88.0082346886 | 4201840640 | 3041189888 | 116916224 / 3758092288 | 149312 / 299516 | 7.0.0-4437-enterprise | +----------------+----------+-----------------+------------+------------+------------------------+-------------------+-----------------------+ 2021-02-11 05:36:42,971 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:43,663 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:44,384 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:45,124 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:45,752 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:46,548 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:47,194 | test | INFO | pool-1-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:47,875 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:48,565 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:49,076 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:49,706 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:50,450 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:51,194 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:51,776 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:52,588 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:53,243 | test | INFO | pool-1-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:53,928 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:54,605 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:55,164 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:55,773 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:56,618 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:57,269 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:57,812 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:58,640 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:36:59,296 | test | INFO | pool-1-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:00,010 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:00,628 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:01,269 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:01,946 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:02,693 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:03,303 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:03,983 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:04,720 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:05,322 | test | INFO | pool-1-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:06,095 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:06,641 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:07,334 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:08,023 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:08,763 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:09,463 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:10,026 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:10,785 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:11,427 | test | INFO | pool-1-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:12,125 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:12,709 | test | INFO | pool-1-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:37:17,716 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4388] Validating collection stats and item counts 2021-02-11 05:37:55,088 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | ephemeral | 2 | none | 0 | 20050 | 314572800 | 89136808 | 102 | | bucket2 | ephemeral | 2 | none | 0 | 15075 | 314572800 | 78653032 | 102 | | default | ephemeral | 2 | none | 0 | 500000 | 4718592000 | 754276120 | 102 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-02-11 05:37:55,091 | test | INFO | MainThread | [basetestcase:log_setup_status:533] ========= CollectionBase setup complete for test #1 test_temp_MB ========= 2021-02-11 05:37:59,687 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 1} 2021-02-11 05:37:59,687 | test | INFO | MainThread | [collections_rebalance:test_temp_MB:850] Cycle 0 2021-02-11 05:37:59,750 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-11 05:37:59,756 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-11 05:38:08,434 | test | INFO | pool-1-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4437-enterprise | 18.3689220479 | Cluster node | | 172.23.105.217 | kv | 7.0.0-4437-enterprise | 17.1487074482 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4437-enterprise | 16.4404609475 | Cluster node | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-11 05:38:08,476 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-11 05:38:15,286 | test | INFO | pool-1-thread-2 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:38:15,289 | test | INFO | pool-1-thread-2 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5233 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:38:15,483 | test | INFO | pool-1-thread-2 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5555 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:38:15,484 | test | INFO | pool-1-thread-2 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5408 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5541 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5598 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5555 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:38:15,625 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-11 05:38:16,757 | test | ERROR | pool-1-thread-6 | [task:call:3555] Failed to load 5 docs from 5000 to 5800 2021-02-11 05:38:19,013 | test | INFO | pool-1-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....**...................**..................................................................**..........X | 22242.449 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-11 05:38:20,292 | test | INFO | pool-1-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................................**.............................................**...................*X | 819.903 | | 449.000 | .....................................................X | 449.000 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-02-11 05:38:20,813 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 4.18383853901 2021-02-11 05:38:21,904 | test | INFO | pool-1-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.......**..............................**...............................................................*X | 1131.826 | | 914.200 | ...............................................................................*X | 914.200 | +---------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-02-11 05:38:25,869 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 9.32313033419 2021-02-11 05:38:30,921 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 14.1759235094 2021-02-11 05:38:35,983 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 16.9863669888 2021-02-11 05:38:41,026 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 19.796022388 2021-02-11 05:38:46,078 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 22.9367836893 2021-02-11 05:38:51,132 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 26.0281910018 2021-02-11 05:38:56,178 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 29.5155561021 2021-02-11 05:39:01,253 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 32.4889861149 2021-02-11 05:39:06,332 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 34.5640135239 2021-02-11 05:39:11,418 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 39.7228498566 2021-02-11 05:39:16,473 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 44.6132726824 2021-02-11 05:39:21,522 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 48.3312029473 2021-02-11 05:39:26,569 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 51.5131354192 2021-02-11 05:39:31,644 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 54.5915761217 2021-02-11 05:39:36,691 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 57.7701406915 2021-02-11 05:39:41,742 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 60.9204712459 2021-02-11 05:39:46,789 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 63.9197869047 2021-02-11 05:39:51,832 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-11 05:39:56,874 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 68.5963540829 2021-02-11 05:40:01,944 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 72.3682499438 2021-02-11 05:40:07,019 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 76.5684412362 2021-02-11 05:40:12,065 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 78.7212761798 2021-02-11 05:40:17,144 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 82.6055362055 2021-02-11 05:40:22,223 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 84.7550032471 2021-02-11 05:40:27,267 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 86.577006543 2021-02-11 05:40:32,328 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 88.6067227845 2021-02-11 05:40:37,428 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 90.6281356525 2021-02-11 05:40:42,486 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 92.7833934941 2021-02-11 05:40:47,553 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 94.9685731092 2021-02-11 05:40:52,609 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 97.0841106052 2021-02-11 05:40:57,681 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: running, progress: 99.2321814234 2021-02-11 05:41:07,775 | test | INFO | pool-1-thread-21 | [task:check:322] Rebalance - status: none, progress: 100 2021-02-11 05:41:07,815 | test | INFO | pool-1-thread-21 | [task:check:379] Rebalance completed with progress: 100% in 179.381000042 sec 2021-02-11 05:41:07,839 | test | WARNING | Thread-611 | [bucket_ready_functions:validate_crud_task_per_collection:500] bucket1:scope2:collection2 create failed keys from task: [u'test_collections-5408', u'test_collections-5233', u'test_collections-5541', u'test_collections-5598', u'test_collections-5555'] 2021-02-11 05:41:09,559 | test | INFO | pool-1-thread-12 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:10,272 | test | INFO | pool-1-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:10,868 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:11,664 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:12,318 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:13,025 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:13,780 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:14,482 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:15,187 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:15,358 | test | INFO | pool-1-thread-12 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:15,990 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:16,088 | test | INFO | pool-1-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:16,634 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:16,641 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:17,413 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:17,440 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:18,122 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:18,242 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:18,871 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:18,905 | test | INFO | pool-1-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:19,506 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:19,562 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:20,263 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:20,937 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:21,197 | test | INFO | pool-1-thread-12 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:21,785 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:21,841 | test | INFO | pool-1-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:22,490 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:22,507 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:23,207 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:23,239 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:23,888 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:24,078 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:24,631 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:24,724 | test | INFO | pool-1-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:25,289 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:25,316 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:26,078 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:26,697 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:26,959 | test | INFO | pool-1-thread-12 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:27,586 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:27,693 | test | INFO | pool-1-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:28,299 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:28,319 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:29,035 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:29,072 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:29,644 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:29,880 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:30,375 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:30,538 | test | INFO | pool-1-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:30,996 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:31,121 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:31,864 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:32,426 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:32,778 | test | INFO | pool-1-thread-12 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:33,407 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:33,484 | test | INFO | pool-1-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:34,134 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:34,154 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:34,806 | test | INFO | pool-1-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:34,946 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:35,372 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:35,759 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:36,164 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:36,331 | test | INFO | pool-1-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:36,719 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:36,881 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:37,638 | test | INFO | pool-1-thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:38,180 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:39,217 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:39,967 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:40,762 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:41,601 | test | INFO | pool-1-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:42,141 | test | INFO | pool-1-thread-15 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:42,700 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:41:47,700 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4388] Validating collection stats and item counts 2021-02-11 05:42:47,523 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-11 05:42:47,529 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-11 05:42:47,647 | test | INFO | pool-1-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4437-enterprise | 14.856557377 | Cluster node | | 172.23.106.237 | kv | 7.0.0-4437-enterprise | 13.717491076 | --- OUT ---> | | 172.23.105.220 | kv | 7.0.0-4437-enterprise | 13.9059304703 | --- OUT ---> | | 172.23.105.217 | kv | 7.0.0-4437-enterprise | 16.602711691 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4437-enterprise | 12.4648786718 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-11 05:42:47,663 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-11 05:42:51,338 | test | INFO | pool-1-thread-10 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2403 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:51,505 | test | INFO | pool-1-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:51,562 | test | INFO | pool-1-thread-10 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2593 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:51,686 | test | INFO | pool-1-thread-29 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2464 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:51,994 | test | ERROR | pool-1-thread-1 | [task:call:3555] Failed to load 1 docs from 1960 to 2881 2021-02-11 05:42:51,997 | test | INFO | pool-1-thread-29 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2813 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:52,112 | test | ERROR | pool-1-thread-1 | [task:call:3555] Failed to load 3 docs from 1960 to 2881 2021-02-11 05:42:52,507 | test | ERROR | pool-1-thread-1 | [task:call:3555] Failed to load 2 docs from 1960 to 2881 2021-02-11 05:42:52,924 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 0.649447305923 2021-02-11 05:42:53,150 | test | INFO | pool-1-thread-28 | [table_view:display:72] Keys failed in bucket1:scope2:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1172 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:54,240 | test | ERROR | pool-1-thread-1 | [task:call:3555] Failed to load 1 docs from 1000 to 1960 2021-02-11 05:42:54,480 | test | INFO | pool-1-thread-17 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1469 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:54,688 | test | INFO | pool-1-thread-17 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1904 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-1894 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:55,276 | test | ERROR | pool-1-thread-1 | [task:call:3555] Failed to load 3 docs from 1000 to 1960 2021-02-11 05:42:57,996 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 2.03194728029 2021-02-11 05:42:58,427 | test | INFO | pool-1-thread-2 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:58,427 | test | INFO | pool-1-thread-2 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6418 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:42:59,107 | test | ERROR | pool-1-thread-1 | [task:call:3555] Failed to load 1 docs from 5800 to 6568 2021-02-11 05:43:02,776 | test | INFO | pool-1-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........................................*..........................**..................................X | 15584.300 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-11 05:43:03,038 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 3.77193163497 2021-02-11 05:43:04,118 | test | INFO | pool-1-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...............................*..................*......................*.............................X | 704.712 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-02-11 05:43:04,628 | test | INFO | pool-1-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **......................................*........................*...........................*...........X | 974.475 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-02-11 05:43:08,112 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 5.85346505687 2021-02-11 05:43:13,164 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 8.23661938014 2021-02-11 05:43:18,237 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 10.5908412832 2021-02-11 05:43:23,279 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 12.9906003256 2021-02-11 05:43:28,328 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 16.6029781918 2021-02-11 05:43:33,372 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 20.0340457929 2021-02-11 05:43:38,421 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 23.9137619567 2021-02-11 05:43:43,460 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 27.8204055878 2021-02-11 05:43:48,503 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 31.5277105843 2021-02-11 05:43:53,551 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-11 05:43:58,607 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 33.3333333333 2021-02-11 05:44:03,663 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 33.9827806393 2021-02-11 05:44:08,713 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 37.1794413076 2021-02-11 05:44:13,759 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 39.4376144778 2021-02-11 05:44:18,795 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 41.7091158018 2021-02-11 05:44:23,835 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 44.0643176443 2021-02-11 05:44:28,878 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 46.7607320756 2021-02-11 05:44:33,920 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 50.1035222502 2021-02-11 05:44:38,960 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 53.7848065093 2021-02-11 05:44:44,013 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 57.5701239046 2021-02-11 05:44:49,062 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 61.7394314973 2021-02-11 05:44:54,101 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 65.0846637639 2021-02-11 05:44:59,138 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-11 05:45:04,177 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-11 05:45:09,219 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 67.4090286418 2021-02-11 05:45:14,256 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 69.2476607452 2021-02-11 05:45:19,302 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 71.2053584769 2021-02-11 05:45:24,345 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 72.6035634682 2021-02-11 05:45:29,388 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 74.0910915267 2021-02-11 05:45:34,420 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 75.6280339086 2021-02-11 05:45:39,486 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 77.1166537565 2021-02-11 05:45:44,529 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 78.5419097511 2021-02-11 05:45:49,569 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 80.4683245351 2021-02-11 05:45:54,625 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 82.9890488885 2021-02-11 05:45:59,658 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 85.0238295037 2021-02-11 05:46:04,696 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 87.6488946359 2021-02-11 05:46:09,746 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 90.2542248045 2021-02-11 05:46:14,786 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 92.840047643 2021-02-11 05:46:19,832 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 95.392214604 2021-02-11 05:46:24,869 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 97.7210075851 2021-02-11 05:46:29,914 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 99.3732193732 2021-02-11 05:46:34,967 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-11 05:46:40,012 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: running, progress: 100 2021-02-11 05:46:50,071 | test | INFO | pool-1-thread-22 | [task:check:322] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.105.220:8091 during connect (<_realsocket at 0x26826 type=client open_count=1 channel=[id: 0x4b55db7c, 0.0.0.0/0.0.0.0:53779] timeout=120.0>) 2021-02-11 05:46:50,107 | test | ERROR | pool-1-thread-22 | [rest_client:_http_request:751] Socket error while connecting to http://172.23.105.220:8091/nodes/self error [Errno 111] Connection refused 2021-02-11 05:46:52,183 | test | INFO | pool-1-thread-22 | [task:check:379] Rebalance completed with progress: 100% in 244.550000191 sec 2021-02-11 05:46:52,201 | test | WARNING | Thread-853 | [bucket_ready_functions:validate_crud_task_per_collection:500] bucket1:scope2:collection1 update failed keys from task: [u'test_collections-2593'] 2021-02-11 05:46:52,201 | test | WARNING | Thread-854 | [bucket_ready_functions:validate_crud_task_per_collection:500] bucket1:scope2:collection2 update failed keys from task: [u'test_collections-2813', u'test_collections-2464'] 2021-02-11 05:46:52,201 | test | WARNING | Thread-851 | [bucket_ready_functions:validate_crud_task_per_collection:500] bucket1:scope1:collection1 update failed keys from task: [u'test_collections-2403', u'test_collections-2371', u'test_collections-2593'] 2021-02-11 05:46:52,210 | test | WARNING | Thread-853 | [bucket_ready_functions:validate_crud_task_per_collection:500] bucket1:scope2:collection1 delete failed keys from task: [u'test_collections-1172'] 2021-02-11 05:46:52,220 | test | WARNING | Thread-854 | [bucket_ready_functions:validate_crud_task_per_collection:500] bucket1:scope2:collection2 delete failed keys from task: [u'test_collections-1904', u'test_collections-1469', u'test_collections-1894'] 2021-02-11 05:46:52,221 | test | WARNING | Thread-851 | [bucket_ready_functions:validate_crud_task_per_collection:500] bucket1:scope1:collection1 create failed keys from task: [u'test_collections-6418'] 2021-02-11 05:46:54,230 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:55,125 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:55,645 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:56,408 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:57,168 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:57,747 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:58,588 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:59,263 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:46:59,822 | test | INFO | pool-1-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:00,423 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:01,243 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:01,875 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:02,585 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:03,328 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:03,964 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:04,766 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:05,375 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:05,921 | test | INFO | pool-1-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:06,572 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:07,467 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:07,994 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:08,743 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:09,571 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:10,055 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:10,992 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:11,529 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:12,051 | test | INFO | pool-1-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:12,696 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:13,723 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:14,276 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:14,937 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:15,693 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:16,150 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:17,157 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:17,687 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:18,174 | test | INFO | pool-1-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:18,818 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:19,861 | test | INFO | pool-1-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:20,476 | test | INFO | pool-1-thread-20 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:21,062 | test | INFO | pool-1-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:21,880 | test | INFO | pool-1-thread-11 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:22,278 | test | INFO | pool-1-thread-18 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:23,266 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:23,901 | test | INFO | pool-1-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:24,338 | test | INFO | pool-1-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-02-11 05:47:29,339 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4388] Validating collection stats and item counts 2021-02-11 05:48:06,131 | test | INFO | MainThread | [collections_rebalance:test_temp_MB:850] Cycle 1 2021-02-11 05:48:06,201 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4631] Performing scope/collection specific operations 2021-02-11 05:48:06,207 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4721] Done Performing scope/collection specific operations 2021-02-11 05:48:14,161 | test | INFO | pool-1-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.105.215 | kv | 7.0.0-4437-enterprise | 23.736207339 | Cluster node | | 172.23.105.217 | kv | 7.0.0-4437-enterprise | 20.6993363961 | Cluster node | | 172.23.105.219 | kv | 7.0.0-4437-enterprise | 20.7320194523 | Cluster node | | 172.23.105.220 | None | | | <--- IN --- | | 172.23.106.237 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-02-11 05:48:14,184 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 0 2021-02-11 05:48:20,040 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 0.0463229477015 2021-02-11 05:48:21,190 | test | INFO | pool-1-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2880 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:48:22,448 | test | INFO | pool-1-thread-22 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:48:22,448 | test | INFO | pool-1-thread-22 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7234 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:48:23,075 | test | ERROR | pool-1-thread-13 | [task:call:3555] Failed to load 1 docs from 1960 to 2881 2021-02-11 05:48:23,273 | test | INFO | pool-1-thread-24 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:48:23,273 | test | INFO | pool-1-thread-24 | [table_view:display:72] Keys failed in bucket1:scope2:collection2 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7171 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-02-11 05:48:23,792 | test | ERROR | pool-1-thread-13 | [task:call:3555] Failed to load 1 docs from 6568 to 7305 2021-02-11 05:48:24,072 | test | ERROR | pool-1-thread-13 | [task:call:3555] Failed to load 1 docs from 6568 to 7305 2021-02-11 05:48:25,187 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 2.01276799178 2021-02-11 05:48:26,767 | test | INFO | pool-1-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.......................................*.......................................***......................X | 17658.900 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-02-11 05:48:28,618 | test | INFO | pool-1-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***................................................................**...................................**.X | 662.600 | | 332.200 | .................................................X | 332.200 | | 209.600 | ..............................X | 209.600 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-02-11 05:48:30,190 | test | INFO | pool-1-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.............................................*............................................***...........X | 627.500 | | 543.400 | .....................................................................................*.X | 548.000 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-02-11 05:48:30,250 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 6.61540234363 2021-02-11 05:48:35,322 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 11.6784977 2021-02-11 05:48:40,371 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 16.5163479203 2021-02-11 05:48:45,438 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 19.1820396231 2021-02-11 05:48:50,489 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 22.4606369514 2021-02-11 05:48:55,530 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 25.8722346576 2021-02-11 05:49:00,595 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 29.1892549694 2021-02-11 05:49:05,647 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 32.8301886792 2021-02-11 05:49:10,700 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 35.3928666646 2021-02-11 05:49:15,765 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 40.2832894903 2021-02-11 05:49:20,838 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 45.3983661172 2021-02-11 05:49:25,898 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 50.4521936495 2021-02-11 05:49:30,963 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 53.8501008193 2021-02-11 05:49:36,022 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 57.0645705466 2021-02-11 05:49:41,073 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 60.3675398531 2021-02-11 05:49:46,138 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 63.7625366594 2021-02-11 05:49:51,203 | test | INFO | pool-1-thread-5 | [task:check:322] Rebalance - status: running, progress: 66.6666666667 2021-02-11 05:49:56,259 | test | ERROR | pool-1-thread-5 | [rest_client:_rebalance_status_and_progress:1485] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed 2021-02-11 05:49:56,301 | test | INFO | pool-1-thread-5 | [rest_client:print_UI_logs:2580] Latest logs from UI on 172.23.105.215: 2021-02-11 05:49:56,302 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613051393568L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:53.568Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.6154.20>,\n {{{{{badmatch,\n [{<27502.32461.5>,\n {done,exit,\n {socket_closed,\n {gen_server,call,\n [<27502.32339.5>,\n {setup_streams,[508,509,510,511]},\n infinity]}},\n [{gen_server,call,3,\n [{file,"gen_server.erl"},\n {line,223}]},\n {dcp_replicator,\n \'-spawn_and_wait/1-fun-0-\',1,\n [{file,"src/dcp_replicator.erl"},\n {line,243}]}]}}]},\n [{misc,\n sync_shutdown_many_i_am_trapping_exits,\n 1,\n [{file,"src/misc.erl"},{line,1415}]},\n {dcp_replicator,spawn_and_wait,1,\n [{file,"src/dcp_replicator.erl"},\n {line,264}]},\n {dcp_replicator,handle_call,3,\n [{file,"src/dcp_replicator.erl"},\n {line,121}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},{line,661}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},{line,690}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,249}]}]},\n {gen_server,call,\n [<27502.32338.5>,get_partitions,\n infinity]}},\n {gen_server,call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,1020},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.219\'},\n {if_rebalance,<0.5106.20>,\n {update_vbucket_state,817,active,paused,\n undefined,\n [[\'ns_1@172.23.105.219\',\n \'ns_1@172.23.105.215\',\n \'ns_1@172.23.105.217\'],\n [\'ns_1@172.23.105.217\',\n \'ns_1@172.23.106.237\',\n \'ns_1@172.23.105.219\']]}},\n infinity]}}}}}.\nRebalance Operation Id = 4cbf8bf16669a385e03d5a1ce6f2ecb1'} 2021-02-11 05:49:56,302 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613051393441L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:53.441Z', u'text': u'Worker <0.5996.20> (for action {move,{817,\n [\'ns_1@172.23.105.219\',\n \'ns_1@172.23.105.215\',\n \'ns_1@172.23.105.217\'],\n [\'ns_1@172.23.105.217\',\n \'ns_1@172.23.106.237\',\n \'ns_1@172.23.105.219\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.6154.20>,\n {{{{{badmatch,\n [{<27502.32461.5>,\n {done,\n exit,\n {socket_closed,\n {gen_server,\n call,\n [<27502.32339.5>,\n {setup_streams,\n [508,\n 509,\n 510,\n 511]},\n infinity]}},\n [{gen_server,\n call,\n 3,\n [{file,\n "gen_server.erl"},\n {line,\n 223}]},\n {dcp_replicator,\n \'-spawn_and_wait/1-fun-0-\',\n 1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 243}]}]}}]},\n [{misc,\n sync_shutdown_many_i_am_trapping_exits,\n 1,\n [{file,\n "src/misc.erl"},\n {line,\n 1415}]},\n {dcp_replicator,\n spawn_and_wait,\n 1,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 264}]},\n {dcp_replicator,\n handle_call,\n 3,\n [{file,\n "src/dcp_replicator.erl"},\n {line,\n 121}]},\n {gen_server,\n try_handle_call,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 661}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 690}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 249}]}]},\n {gen_server,\n call,\n [<27502.32338.5>,\n get_partitions,\n infinity]}},\n {gen_server,\n call,\n [\'dcp_replication_manager-default\',\n {get_replicator_pid,\n 1020},\n infinity]}},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.219\'},\n {if_rebalance,\n <0.5106.20>,\n {update_vbucket_state,\n 817,\n active,\n paused,\n undefined,\n [[\'ns_1@172.23.105.219\',\n \'ns_1@172.23.105.215\',\n \'ns_1@172.23.105.217\'],\n [\'ns_1@172.23.105.217\',\n \'ns_1@172.23.106.237\',\n \'ns_1@172.23.105.219\']]}},\n infinity]}}}}'} 2021-02-11 05:49:56,305 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613051392214L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:52.214Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2021-02-11 05:49:56,305 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.237', u'tstamp': 1613051390692L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:50.692Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.237\' in 0 seconds.'} 2021-02-11 05:49:56,305 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.220', u'tstamp': 1613051390689L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:50.689Z', u'text': u'Bucket "default" loaded on node \'ns_1@172.23.105.220\' in 0 seconds.'} 2021-02-11 05:49:56,305 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613051390601L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:50.601Z', u'text': u'Started rebalancing bucket default'} 2021-02-11 05:49:56,306 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613051348299L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:08.299Z', u'text': u'Bucket "bucket2" rebalance does not seem to be swap rebalance'} 2021-02-11 05:49:56,306 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.105.220', u'tstamp': 1613051346741L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:06.741Z', u'text': u'Bucket "bucket2" loaded on node \'ns_1@172.23.105.220\' in 0 seconds.'} 2021-02-11 05:49:56,306 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.237', u'tstamp': 1613051346735L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:06.735Z', u'text': u'Bucket "bucket2" loaded on node \'ns_1@172.23.106.237\' in 0 seconds.'} 2021-02-11 05:49:56,308 | test | ERROR | pool-1-thread-5 | [rest_client:print_UI_logs:2582] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.105.215', u'tstamp': 1613051346622L, u'shortText': u'message', u'serverTime': u'2021-02-11T05:49:06.622Z', u'text': u'Started rebalancing bucket bucket2'} 2021-02-11 05:49:56,308 | test | ERROR | pool-1-thread-5 | [task:call:237] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'status': u'none'} - rebalance failed FAIL 2021-02-11 05:49:56,456 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | ephemeral | 2 | none | 0 | 17743 | 524288000 | 106603880 | 170 | | bucket2 | ephemeral | 2 | none | 0 | 13260 | 524288000 | 95637480 | 170 | | default | ephemeral | 2 | none | 0 | 442100 | 7864320000 | 908663352 | 102 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ ====================================================================== FAIL: test_temp_MB (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 857, in test_temp_MB self.wait_for_rebalance_to_complete(operation) File "pytests/bucket_collections/collections_rebalance.py", line 706, in wait_for_rebalance_to_complete self.assertTrue(task.result, "Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 942.818s FAILED (failures=1) During the test, Remote Connections: 211, Disconnections: 211 SDK Connections: 30, Disconnections: 0 !!!!!! CRITICAL :: SDK disconnection mismatch !!!!! summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 1 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_temp_MB testrunner logs, diags and results are available under /data/workspace/rebalance_sumedh_temp/logs/testrunner-21-Feb-11_05-34-11/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 15m 52s 2 actionable tasks: 2 executed Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 7.0.0-4437,product=cb,parallel=True' [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE