Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s72001 (P0 jython_slave PY3P0 xattr-subdoc-andrei) in workspace /data/workspace/temp_rebalance_even [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_rebalance_even # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 7e951c6259c0b06da0748ca0e515e2a44aa91e6f (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 7e951c6259c0b06da0748ca0e515e2a44aa91e6f > /usr/bin/git rev-list 7e951c6259c0b06da0748ca0e515e2a44aa91e6f # timeout=10 [temp_rebalance_even] $ /bin/sh -xe /tmp/jenkins4930357643868767687.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 #data_path:/data/dta #index_path:/data/idx #cbas_path:['\''/data/d1'\'','\''/data/d2'\'','\''/data/d3'\'','\''/data/d4'\'','\''/data/d5'\'','\''/data/d6'\''] [membase] rest_username:Administrator rest_password:password [cluster1] 1:_1 2:_2 [cluster2] 1:_3 2:_4 [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 #6:_6 [_6] ip:172.23.98.196 port:8091 services:kv,index,n1ql [_2] ip:172.23.98.195 port:8091 services:kv,n1ql [_3] ip:172.23.121.10 port:8091 services:kv [_4] ip:172.23.104.186 port:8091 services:cbas [_5] ip:172.23.120.201 [_1] ip:172.23.120.206' + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 #6:_6 [_1] ip:172.23.98.196 [_2] ip:172.23.98.195 [_3] ip:172.23.121.10 [_4] ip:172.23.104.186 [_6] ip:172.23.120.201 [_5] ip:172.23.120.206' + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From git://github.com/couchbaselabs/TAF * branch master -> FETCH_HEAD Already up-to-date. + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + guides/gradlew check -s Starting a Gradle Daemon, 139 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 > Task :compileJava Note: /data/workspace/temp_rebalance_even/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :processResources > Task :classes > Task :compileTestJava NO-SOURCE > Task :processTestResources NO-SOURCE > Task :testClasses UP-TO-DATE > Task :test NO-SOURCE > Task :check UP-TO-DATE Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 17s 2 actionable tasks: 2 executed + guides/gradlew -v ------------------------------------------------------------ Gradle 6.2.2 ------------------------------------------------------------ Build time: 2020-03-04 08:49:31 UTC Revision: 7d0bf6dcb46c143bcc3b7a0fa40a8e5ca28e5856 Kotlin: 1.3.61 Groovy: 2.5.8 Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 JVM: 1.8.0_252 (Oracle Corporation 25.252-b09) OS: Linux 3.10.0-693.2.2.el7.x86_64 amd64 + guides/gradlew install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini -p version=7.0.0-5085,product=cb,parallel=,init_nodes=True,url=' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-5085,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-5085,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-5085,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-5085,product=cb,parallel=,init_nodes=True,url= Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops.ini -p version=7.0.0-5085,product=cb,parallel=,init_nodes=True,url= > Task :install 2021-05-02 21:11:23,348 - root - WARNING - URL: is invalid, will use version to locate build 2021-05-02 21:12:33,786 - root - INFO - Done with uninstall on 172.23.98.196. 2021-05-02 21:12:36,698 - root - INFO - Done with uninstall on 172.23.120.206. 2021-05-02 21:12:38,270 - root - INFO - Done with uninstall on 172.23.121.10. 2021-05-02 21:12:42,525 - root - INFO - Done with uninstall on 172.23.104.186. 2021-05-02 21:12:43,671 - root - INFO - Done with uninstall on 172.23.98.195. 2021-05-02 21:17:31,279 - root - INFO - Done with install on 172.23.120.206. 2021-05-02 21:18:34,033 - root - INFO - Setting KV memory quota as 1886 MB on 172.23.120.206 2021-05-02 21:18:34,282 - root - INFO - Done with init on 172.23.120.206. 2021-05-02 21:18:34,303 - root - INFO - Done with cleanup on 172.23.120.206. 2021-05-02 21:18:37,385 - root - INFO - Done with install on 172.23.98.196. 2021-05-02 21:18:42,339 - root - INFO - Done with install on 172.23.121.10. 2021-05-02 21:18:45,253 - root - INFO - Done with install on 172.23.104.186. 2021-05-02 21:18:45,970 - root - INFO - Done with install on 172.23.98.195. 2021-05-02 21:19:39,213 - root - INFO - Setting KV memory quota as 1887 MB on 172.23.98.196 2021-05-02 21:19:39,494 - root - INFO - Done with init on 172.23.98.196. 2021-05-02 21:19:39,581 - root - INFO - Done with cleanup on 172.23.98.196. 2021-05-02 21:19:44,194 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.121.10 2021-05-02 21:19:44,477 - root - INFO - Done with init on 172.23.121.10. 2021-05-02 21:19:44,546 - root - INFO - Done with cleanup on 172.23.121.10. 2021-05-02 21:19:46,967 - root - INFO - Setting KV memory quota as 3106 MB on 172.23.104.186 2021-05-02 21:19:47,220 - root - INFO - Done with init on 172.23.104.186. 2021-05-02 21:19:47,467 - root - INFO - Done with cleanup on 172.23.104.186. 2021-05-02 21:19:47,762 - root - INFO - Setting KV memory quota as 1887 MB on 172.23.98.195 2021-05-02 21:19:48,022 - root - INFO - Done with init on 172.23.98.195. 2021-05-02 21:19:48,111 - root - INFO - Done with cleanup on 172.23.98.195. 2021-05-02 21:20:02,447 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-02 21:20:02,490 - root - INFO - node:172.23.98.196:8091 version:7.0.0-5085-enterprise aFamily:inet services:[u'kv'] 2021-05-02 21:20:02,526 - root - INFO - node:172.23.98.195:8091 version:7.0.0-5085-enterprise aFamily:inet services:[u'kv'] 2021-05-02 21:20:02,562 - root - INFO - node:172.23.121.10:8091 version:7.0.0-5085-enterprise aFamily:inet services:[u'kv'] 2021-05-02 21:20:02,599 - root - INFO - node:172.23.104.186:8091 version:7.0.0-5085-enterprise aFamily:inet services:[u'kv'] 2021-05-02 21:20:02,635 - root - INFO - node:172.23.120.206:8091 version:7.0.0-5085-enterprise aFamily:inet services:[u'kv'] 2021-05-02 21:20:02,638 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-02 21:20:02,638 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-02 21:20:02,640 - root - INFO - INSTALL COMPLETED ON: 172.23.98.196 2021-05-02 21:20:02,641 - root - INFO - INSTALL COMPLETED ON: 172.23.98.195 2021-05-02 21:20:02,641 - root - INFO - INSTALL COMPLETED ON: 172.23.121.10 2021-05-02 21:20:02,642 - root - INFO - INSTALL COMPLETED ON: 172.23.104.186 2021-05-02 21:20:02,644 - root - INFO - INSTALL COMPLETED ON: 172.23.120.206 2021-05-02 21:20:02,644 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-05-02 21:20:02,653 - root - INFO - TOTAL INSTALL TIME = 519.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 8m 53s 1 actionable task: 1 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_rebalance_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'cluster_name': 'win10-bucket-ops', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops.ini', 'num_nodes': 5, 'quota_percent': '95', 'rerun': 'False', 'spec': 'collections_rebalance_crud_on_collections'} Only cases in GROUPs 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' will be executed Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_1', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 1, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-02 21:20:29,971 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-02 21:20:31,265 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-02 21:20:33,364 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xd4913a3b, 0.0.0.0/0.0.0.0:44283] timeout=300.0>) 2021-05-02 21:20:38,921 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x65740839, 0.0.0.0/0.0.0.0:42646] timeout=300.0>) 2021-05-02 21:20:42,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0xab4e6049, 0.0.0.0/0.0.0.0:46744] timeout=300.0>) 2021-05-02 21:20:44,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xbe82507b, 0.0.0.0/0.0.0.0:40439] timeout=300.0>) 2021-05-02 21:20:46,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x64f4aeef, 0.0.0.0/0.0.0.0:53689] timeout=300.0>) 2021-05-02 21:20:48,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xe48c8702, 0.0.0.0/0.0.0.0:33944] timeout=300.0>) 2021-05-02 21:20:50,358 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x3c368a07, 0.0.0.0/0.0.0.0:53437] timeout=300.0>) 2021-05-02 21:20:52,398 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0xdb91a07c, 0.0.0.0/0.0.0.0:38179] timeout=300.0>) 2021-05-02 21:20:54,440 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0xd4382c2c, 0.0.0.0/0.0.0.0:35782] timeout=300.0>) 2021-05-02 21:21:02,065 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x35289ab7, 0.0.0.0/0.0.0.0:35839] timeout=300.0>) 2021-05-02 21:21:04,108 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0xf156fa29, 0.0.0.0/0.0.0.0:34773] timeout=300.0>) 2021-05-02 21:21:06,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x4b2e281e, 0.0.0.0/0.0.0.0:37133] timeout=300.0>) 2021-05-02 21:21:08,188 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0xbc83098a, 0.0.0.0/0.0.0.0:48015] timeout=300.0>) 2021-05-02 21:21:10,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x490a7e70, 0.0.0.0/0.0.0.0:58336] timeout=300.0>) 2021-05-02 21:21:12,262 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x6d9b8477, 0.0.0.0/0.0.0.0:57397] timeout=300.0>) 2021-05-02 21:21:14,295 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x7e650a77, 0.0.0.0/0.0.0.0:40257] timeout=300.0>) 2021-05-02 21:21:16,325 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x20616774, 0.0.0.0/0.0.0.0:58498] timeout=300.0>) 2021-05-02 21:21:18,361 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xbb7c3556, 0.0.0.0/0.0.0.0:59199] timeout=300.0>) 2021-05-02 21:21:25,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x3c01a227, 0.0.0.0/0.0.0.0:35830] timeout=300.0>) 2021-05-02 21:21:27,855 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0xfb0ace13, 0.0.0.0/0.0.0.0:47760] timeout=300.0>) 2021-05-02 21:21:29,891 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0x9b7bfd4a, 0.0.0.0/0.0.0.0:56819] timeout=300.0>) 2021-05-02 21:21:31,927 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0xd98ff5a1, 0.0.0.0/0.0.0.0:34989] timeout=300.0>) 2021-05-02 21:21:33,959 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0x6a4eab64, 0.0.0.0/0.0.0.0:41270] timeout=300.0>) 2021-05-02 21:21:35,989 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0xa79629e8, 0.0.0.0/0.0.0.0:51530] timeout=300.0>) 2021-05-02 21:21:38,017 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0x99d3db93, 0.0.0.0/0.0.0.0:38965] timeout=300.0>) 2021-05-02 21:21:40,052 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0x912f6142, 0.0.0.0/0.0.0.0:39281] timeout=300.0>) 2021-05-02 21:21:42,085 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0x5942af23, 0.0.0.0/0.0.0.0:46314] timeout=300.0>) 2021-05-02 21:21:44,117 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0xb92174ee, 0.0.0.0/0.0.0.0:47092] timeout=300.0>) 2021-05-02 21:21:52,068 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0x0375b463, 0.0.0.0/0.0.0.0:38963] timeout=300.0>) 2021-05-02 21:21:54,101 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2a type=client open_count=1 channel=[id: 0x43486839, 0.0.0.0/0.0.0.0:42698] timeout=300.0>) 2021-05-02 21:21:56,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2b type=client open_count=1 channel=[id: 0x038bc37e, 0.0.0.0/0.0.0.0:42789] timeout=300.0>) 2021-05-02 21:21:58,167 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2c type=client open_count=1 channel=[id: 0xd9f8bf3f, 0.0.0.0/0.0.0.0:43118] timeout=300.0>) 2021-05-02 21:22:00,203 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2d type=client open_count=1 channel=[id: 0x6650dfbe, 0.0.0.0/0.0.0.0:51456] timeout=300.0>) 2021-05-02 21:22:02,233 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2e type=client open_count=1 channel=[id: 0x2d487895, 0.0.0.0/0.0.0.0:38182] timeout=300.0>) 2021-05-02 21:22:04,255 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2f type=client open_count=1 channel=[id: 0xe8e1c51a, 0.0.0.0/0.0.0.0:48469] timeout=300.0>) 2021-05-02 21:22:06,282 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x30 type=client open_count=1 channel=[id: 0x5bb9ecba, 0.0.0.0/0.0.0.0:59077] timeout=300.0>) 2021-05-02 21:22:08,318 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x31 type=client open_count=1 channel=[id: 0x3b273722, 0.0.0.0/0.0.0.0:42927] timeout=300.0>) 2021-05-02 21:22:10,349 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x32 type=client open_count=1 channel=[id: 0x1196c98a, 0.0.0.0/0.0.0.0:52176] timeout=300.0>) 2021-05-02 21:22:15,046 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x33 type=client open_count=1 channel=[id: 0x332d1339, 0.0.0.0/0.0.0.0:53579] timeout=300.0>) 2021-05-02 21:22:17,078 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x34 type=client open_count=1 channel=[id: 0x76c2de29, 0.0.0.0/0.0.0.0:50966] timeout=300.0>) 2021-05-02 21:22:19,111 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x35 type=client open_count=1 channel=[id: 0x2d91793b, 0.0.0.0/0.0.0.0:49505] timeout=300.0>) 2021-05-02 21:22:21,144 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x36 type=client open_count=1 channel=[id: 0x7e60db34, 0.0.0.0/0.0.0.0:58498] timeout=300.0>) 2021-05-02 21:22:23,167 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x37 type=client open_count=1 channel=[id: 0xe9b4809d, 0.0.0.0/0.0.0.0:50785] timeout=300.0>) 2021-05-02 21:22:25,201 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 21:22:42,013 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-02 21:22:56,446 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-02 21:22:56,476 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-02 21:22:56,528 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-02 21:23:20,572 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 8.85692657078 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-05-02 21:23:20,607 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 21:23:30,717 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-02 21:23:30,752 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1809999943 sec 2021-05-02 21:23:31,177 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-02 21:24:12,288 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-02 21:24:12,591 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 629145600 | 665165712 | 39058117 | | bucket2 | couchbase | 2 | none | 0 | 0 | 629145600 | 664700232 | 67248004 | | default | couchbase | 2 | none | 0 | 0 | 6291456000 | 825109208 | 14916836 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-02 21:24:12,592 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-02 21:24:49,306 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 21:24:49,342 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 21:46:29,759 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **................**................**...**..............*..............................................*X | 17.497 | | 16.002 | ......................................................................................*X | 16.002 | | 2.002 | .........**X | 2.002 | | 0.000 | *******************.............**.................*................*X | 8.502 | | 6.504 | ..................................X | 6.504 | | 4.001 | ....................*X | 4.001 | | 1.500 | .......**.........................................................**..............................*X | 17.498 | | 12.498 | ...................................................................*X | 12.498 | | 8.000 | ..........................................X | 8.000 | | 2.500 | ............*X | 2.500 | | 1.500 | .......X | 1.500 | | 1.000 | ....****...*..*...***...X | 3.000 | | 2.000 | .........**......*.....*..............*X | 6.500 | | 5.500 | .............................**X | 5.500 | | 4.500 | .......................X | 4.500 | | 4.500 | .......................*X | 4.500 | | 2.500 | ............*X | 2.500 | | 1.000 | ....X | 1.000 | | 0.000 | ***************************************.....***...*...........*...........**..........*X | 7.496 | | 5.501 | .............................X | 5.501 | | 3.501 | ..................*X | 3.501 | | 1.501 | .......**................***........*...*...**..**.................................***.................X | 16.508 | | 11.004 | ...........................................................X | 11.004 | | 6.000 | ...............................*X | 6.000 | | 1.000 | ....**...X | 1.500 | | 1.000 | ....**...**........*X | 3.000 | | 2.000 | .........**....................*....................***.................................................*.X | 18.304 | | 17.704 | ...............................................................................................*X | 17.704 | | 8.704 | ..............................................*X | 8.704 | | 4.200 | .....................X | 4.200 | | 0.000 | ******************...................***.....................*...........*......*X | 10.499 | | 7.503 | .......................................*X | 7.503 | | 3.001 | ...............**X | 3.001 | | 3.000 | ...............X | 3.000 | | 2.000 | .........X | 2.000 | | 1.000 | ....*X | 1.000 | | 0.000 | *************..**...*...**X | 1.500 | | 1.000 | ....X | 1.000 | | 0.500 | .X | 0.500 | | 0.000 | *********..***..*...X | 1.400 | | 1.100 | .....*X | 1.100 | | 0.600 | ..*X | 0.600 | | 0.100 | X | 0.100 | | 0.000 | **********************..*...**...*X | 1.500 | | 1.000 | ....X | 1.000 | | 0.500 | .X | 0.500 | | 0.000 | ******************************...................................**..............................**........X | 13.496 | | 9.999 | .....................................................*X | 9.999 | | 4.999 | ..........................*X | 4.999 | | 2.001 | .........X | 2.001 | | 2.000 | .........**.........**........*...*X | 5.499 | | 3.499 | ..................X | 3.499 | | 1.999 | .........X | 1.999 | | 1.000 | ....**...*X | 1.500 | | 1.000 | ....X | 1.000 | | 0.500 | .*X | 0.500 | | 0.000 | ***********..**...**...X | 1.500 | | 1.000 | ....*X | 1.000 | | 0.500 | .**.................*.....................**......................X | 11.501 | | 8.500 | .............................................X | 8.500 | | 4.500 | .......................*X | 4.500 | | 0.500 | .*X | 0.500 | | 0.000 | *********.....*.....**......*X | 2.999 | | 2.001 | .........X | 2.001 | | 1.000 | ....***...................*...................**........*X | 9.498 | | 9.491 | ..................................................**......*X | 10.497 | | 8.497 | .............................................X | 8.497 | | 8.009 | ..........................................X | 8.009 | | 7.502 | .......................................**X | 7.502 | | 4.497 | .......................*X | 4.497 | | 1.500 | .......*X | 1.500 | | 0.000 | *******************************..**......**X | 1.500 | | 1.000 | ....X | 1.000 | | 0.500 | .***...**...X | 1.500 | | 1.000 | ....*X | 1.000 | | 0.500 | .*X | 0.500 | | 0.000 | *************************************.....***...*.....*......**.....*X | 4.500 | | 1.500 | .......*X | 1.500 | | 0.000 | ****************.....**...*X | 1.500 | | 1.000 | ....X | 1.000 | | 0.500 | .*X | 0.500 | | 0.000 | ****................**..............................*..............................*........*X | 15.502 | | 12.008 | ................................................................X | 12.008 | | 8.502 | .............................................*X | 8.502 | | 4.002 | ....................*X | 4.002 | | 2.000 | .........X | 2.000 | | 0.000 | ***..*...**......................**...................................*....................................*X | 17.999 | | 17.501 | ..............................................................................................X | 17.501 | | 11.001 | ...........................................................X | 11.001 | | 4.500 | .......................*X | 4.500 | | 1.500 | .......*X | 1.500 | | 1.000 | ....X | 1.000 | | 0.500 | .*X | 0.500 | | 0.000 | ****..**........*.........**..X | 4.000 | | 2.500 | ............X | 2.500 | | 1.000 | ....*X | 1.000 | | 0.000 | *********************************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------+--------+ 2021-05-02 21:46:31,385 | test | INFO | pool-2-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **..........**..........*...........*X | 9.300 | | 7.799 | .........................**..X | 8.299 | | 8.199 | ...........................*.*X | 8.702 | | 5.202 | ................X | 5.202 | | 2.002 | .....X | 2.002 | | 0.000 | *********************......*...........**............**........**........................**....................................**...X | 29.098 | | 20.098 | ....................................................................*X | 20.098 | | 8.598 | ............................X | 8.598 | | 2.500 | .......*X | 2.500 | | 1.500 | ....**...*....*...*X | 4.499 | | 3.000 | .........X | 3.000 | | 1.500 | ....*X | 1.500 | | 0.000 | ***********.........................*............................*...........................**X | 23.498 | | 16.998 | .........................................................X | 16.998 | | 9.498 | ...............................X | 9.498 | | 2.000 | .....*X | 2.000 | | 1.000 | ..X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | **...***..X | 1.500 | | 0.999 | ..*.....**.......**.........X | 7.106 | | 5.404 | .................X | 5.404 | | 2.904 | ........*X | 2.904 | | 0.400 | *X | 0.400 | | 0.000 | ********...***..X | 1.500 | | 1.000 | ..X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ******......................**..........................*.........................*X | 21.490 | | 17.003 | .........................................................X | 17.003 | | 11.003 | ....................................*X | 11.003 | | 5.503 | .................**.........*.........*.....*X | 11.996 | | 7.496 | ........................*X | 7.496 | | 3.496 | ...........X | 3.496 | | 0.000 | *******************.....**..........*............**.........*X | 10.498 | | 8.502 | ............................*X | 8.502 | | 4.502 | ..............*X | 4.502 | | 2.000 | .....*X | 2.000 | | 0.000 | *****************.....................**..........................**.................X | 18.692 | | 17.297 | ..........................................................*X | 17.297 | | 10.302 | ..................................*X | 10.302 | | 4.802 | ...............X | 4.802 | | 2.500 | .......*..*X | 3.000 | | 2.000 | .....X | 2.000 | | 1.000 | ..*X | 1.000 | | 0.000 | ******.*..***..X | 1.500 | | 1.000 | ..X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | *******...**..**.*..**.......**..*.....****...X | 7.499 | | 6.999 | .......................X | 6.999 | | 4.999 | ................*X | 4.999 | | 3.000 | .........X | 3.000 | | 1.000 | ..***X | 1.000 | | 1.000 | ..*..*X | 1.500 | | 1.000 | ..*X | 1.000 | | 0.000 | ***********************.**..**..X | 1.500 | | 1.000 | ..*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ***.**..*..*X | 1.501 | | 1.001 | ..*X | 1.001 | | 0.501 | X | 0.501 | | 0.000 | *********.*....**...**X | 2.500 | | 2.000 | .....*X | 2.000 | | 1.901 | .....*.....**.....*...X | 5.600 | | 4.100 | .............*X | 4.100 | | 2.100 | ......*X | 2.100 | | 0.500 | X | 0.500 | | 0.000 | ******************.**..**..X | 1.500 | | 1.000 | ..X | 1.000 | | 0.500 | ***..**..*X | 1.500 | | 1.000 | ..X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ********************.*..**..*X | 1.500 | | 1.499 | ....****.*..**..X | 3.001 | | 2.001 | .....X | 2.001 | | 0.999 | ..*X | 0.999 | | 0.000 | *********************************.....*.....**.....X | 4.500 | | 3.000 | .........*X | 3.000 | | 1.500 | ....X | 1.500 | | 0.500 | **..********..X | 1.500 | | 1.000 | ..*****..*X | 1.500 | | 1.000 | ..X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ************......***......X | 3.499 | | 2.999 | .........X | 2.999 | | 2.500 | .......*X | 2.500 | | 1.000 | ..*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | *******......*................*...............**..*X | 11.500 | | 2.500 | .......*X | 2.500 | | 0.000 | **************.**..*..X | 1.500 | | 1.000 | ..*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ****..........***...................*........*.........*X | 13.502 | | 9.002 | .............................X | 9.002 | | 6.006 | ...................*X | 6.006 | | 3.001 | .........X | 3.001 | | 2.001 | .....*X | 2.001 | | 1.000 | ..X | 1.000 | | 0.000 | ******...**..........*...........**...*X | 8.000 | | 2.000 | .....**X | 2.000 | | 0.000 | *********.*....*...**....**...*...*X | 5.500 | | 5.000 | ................X | 5.000 | | 3.000 | .........X | 3.000 | | 1.000 | ..*X | 1.000 | | 0.000 | *****************************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-02 21:46:33,388 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **........**.................*................*..........**........*................**.............**....**X | 725.155 | | 698.532 | ........................................................................................X | 698.532 | | 644.532 | .................................................................................X | 644.532 | | 637.457 | ................................................................................*X | 637.457 | | 584.978 | .........................................................................**.*.**...........*......*.......*X | 781.400 | | 763.600 | ................................................................................................*X | 763.600 | | 723.800 | ...........................................................................................X | 723.800 | | 684.823 | ......................................................................................*X | 684.823 | | 652.559 | ..................................................................................X | 652.559 | | 637.357 | ................................................................................X | 637.357 | | 617.596 | ..............................................................................*X | 617.596 | | 608.021 | ............................................................................*X | 608.021 | | 604.122 | ............................................................................X | 604.122 | | 574.060 | ........................................................................X | 574.060 | | 510.399 | ................................................................*X | 510.399 | | 411.300 | ...................................................*X | 411.300 | | 344.269 | ...........................................X | 344.269 | | 298.469 | .....................................**..*...**....*X | 371.038 | | 343.054 | ..........................................**X | 343.054 | | 332.354 | .........................................X | 332.354 | | 325.543 | ........................................***..**..*..*.**..**.*.**X | 407.300 | | 400.600 | ..................................................*X | 400.600 | | 392.374 | .................................................**.....***....X | 466.032 | | 464.885 | ..........................................................X | 464.885 | | 456.285 | .........................................................*X | 456.285 | | 442.153 | .......................................................**...*..*.**X | 485.580 | | 445.505 | ........................................................X | 445.505 | | 401.761 | ..................................................*X | 401.761 | | 386.361 | ................................................X | 386.361 | | 372.121 | ..............................................X | 372.121 | | 342.269 | ..........................................*X | 342.269 | | 322.845 | ........................................*X | 322.845 | | 317.475 | .......................................*..*X | 330.675 | | 318.711 | .......................................**......***..X | 382.700 | | 371.000 | ..............................................X | 371.000 | | 356.000 | ............................................**....*X | 384.500 | | 363.292 | .............................................*.X | 373.992 | | 340.392 | ..........................................**........**..*.....*X | 456.676 | | 426.605 | .....................................................X | 426.605 | | 400.305 | ..................................................*X | 400.305 | | 370.905 | ..............................................**..*X | 384.595 | | 370.895 | ..............................................*X | 370.895 | | 351.300 | ...........................................*X | 351.300 | | 317.400 | .......................................*.**..*......*..*X | 403.766 | | 403.118 | ..................................................*X | 403.118 | | 373.426 | ..............................................X | 373.426 | | 354.768 | ............................................*X | 354.768 | | 324.692 | ........................................X | 324.692 | | 302.700 | .....................................*X | 302.700 | | 293.000 | ....................................**...**..*X | 332.900 | | 320.800 | ........................................**.X | 328.600 | | 295.300 | ....................................*X | 295.300 | | 287.628 | ...................................X | 287.628 | | 278.828 | ..................................**.**....*..*....*X | 365.898 | | 363.900 | .............................................*X | 363.900 | | 363.100 | .............................................*.**...*...**..*X | 433.700 | | 428.800 | .....................................................****....*.*X | 468.000 | | 455.900 | .........................................................*X | 455.900 | | 440.100 | .......................................................X | 440.100 | | 396.330 | .................................................X | 396.330 | | 371.152 | ..............................................*X | 371.152 | | 324.552 | ........................................*..**.......**.........**X | 467.119 | | 394.600 | .................................................X | 394.600 | | 328.100 | ........................................X | 328.100 | | 306.900 | ......................................**...*X | 330.316 | | 310.336 | ......................................*X | 310.336 | | 287.789 | ...................................**.X | 296.889 | | 295.147 | ....................................***X | 296.678 | | 294.078 | ....................................*.***..X | 312.900 | | 309.900 | ......................................**..**..*..**.**...**.***..***X | 407.241 | | 384.066 | ................................................X | 384.066 | | 361.366 | .............................................*X | 361.366 | | 322.566 | ........................................X | 322.566 | | 308.030 | ......................................X | 308.030 | | 278.330 | ..................................**...***....**..**X | 346.347 | | 331.884 | .........................................X | 331.884 | | 311.484 | ......................................*X | 311.484 | | 311.416 | ......................................X | 311.416 | | 299.835 | .....................................**..**..*......*....**..X | 428.529 | | 417.165 | ....................................................*X | 417.165 | | 373.765 | ..............................................*X | 373.765 | | 343.565 | ..........................................X | 343.565 | | 338.100 | ..........................................**.....**...X | 403.200 | | 398.500 | .................................................X | 398.500 | | 355.300 | ............................................*X | 355.300 | | 310.900 | ......................................X | 310.900 | | 280.600 | ..................................*X | 280.600 | | 273.953 | ..................................**...*.....**....*X | 374.237 | | 370.737 | ..............................................X | 370.737 | | 337.460 | ..........................................X | 337.460 | | 318.195 | .......................................*X | 318.195 | | 316.295 | .......................................*.....****..X | 368.702 | | 341.702 | ..........................................****......**...**....*...**..*X | 477.236 | | 449.952 | ........................................................X | 449.952 | | 411.253 | ...................................................X | 411.253 | | 388.153 | ................................................**.X | 394.100 | | 382.054 | ...............................................*X | 382.054 | | 374.254 | ..............................................*X | 374.254 | | 317.054 | .......................................X | 317.054 | | 302.740 | .....................................X | 302.740 | | 273.785 | ..................................**.......*...**......*X | 402.082 | | 391.782 | .................................................X | 391.782 | | 376.082 | ...............................................*X | 376.082 | | 368.500 | ..............................................*X | 368.500 | | 366.048 | .............................................*X | 366.048 | | 345.748 | ...........................................**..*.**.....*.X | 421.952 | | 420.199 | ....................................................*X | 420.199 | | 405.313 | ..................................................*X | 405.313 | | 395.593 | .................................................X | 395.593 | | 382.093 | ...............................................X | 382.093 | | 340.738 | ..........................................*X | 340.738 | | 337.663 | ..........................................*.**.....*X | 383.650 | | 378.023 | ...............................................X | 378.023 | | 361.821 | .............................................*X | 361.821 | | 341.321 | ..........................................*X | 341.321 | | 335.185 | .........................................*.X | 340.332 | | 324.632 | ........................................**.*...**.......*X | 413.935 | | 407.021 | ...................................................X | 407.021 | | 379.739 | ...............................................X | 379.739 | | 339.139 | ..........................................*X | 339.139 | | 323.900 | ........................................*.**..**......*.....X | 431.586 | | 430.824 | ......................................................**X | 433.624 | | 381.214 | ...............................................*X | 381.214 | | 357.276 | ............................................X | 357.276 | | 296.276 | ....................................X | 296.276 | | 293.981 | ....................................*X | 293.981 | | 263.085 | ................................**..***.......*....*.*X | 371.240 | | 336.240 | ..........................................*X | 336.240 | | 319.052 | .......................................X | 319.052 | | 301.852 | .....................................***..**....*....**.X | 389.226 | | 370.087 | ..............................................*X | 370.087 | | 337.668 | ..........................................***..*........*X | 416.742 | | 409.042 | ...................................................*..*X | 425.742 | | 370.400 | ..............................................**..X | 384.100 | | 357.300 | ............................................***X | 358.800 | | 274.200 | ..................................**X | 274.200 | | 263.200 | ................................*....***.***.**...**.....**X | 367.951 | | 355.051 | ............................................*.X | 364.388 | | 359.486 | .............................................****....**....**...*.**..*X | 476.400 | | 461.600 | ..........................................................X | 461.600 | | 453.054 | ........................................................X | 453.054 | | 422.454 | .....................................................***..***.X | 448.201 | | 390.801 | .................................................X | 390.801 | | 362.635 | .............................................*X | 362.635 | | 350.034 | ...........................................*X | 350.034 | | 344.334 | ...........................................*X | 350.899 | | 324.881 | ........................................***.***...X | 351.900 | | 321.000 | ........................................**X | 324.826 | | 314.926 | .......................................**..*.*X | 339.981 | | 335.481 | .........................................**....***.......*..*..*X | 449.776 | | 433.712 | ......................................................*X | 433.712 | | 406.749 | ...................................................**.*X | 421.038 | | 410.432 | ...................................................*.*X | 419.661 | | 366.361 | .............................................*X | 366.361 | | 294.661 | ....................................**X | 294.661 | | 267.461 | .................................X | 267.461 | | 247.821 | ..............................X | 247.821 | | 235.741 | .............................**.....**.......**X | 333.985 | | 333.285 | .........................................*...X | 354.000 | | 347.200 | ...........................................*X | 347.200 | | 330.900 | .........................................*X | 330.900 | | 316.300 | .......................................*..**....X | 361.975 | | 356.475 | ............................................X | 356.475 | | 349.675 | ...........................................****.....*..**......**...*.....X | 509.143 | | 492.043 | .............................................................*X | 492.043 | | 478.843 | ............................................................*X | 478.843 | | 420.764 | ....................................................X | 420.764 | | 383.164 | ................................................*X | 383.164 | | 338.064 | ..........................................X | 338.064 | | 297.924 | .....................................X | 297.924 | | 271.924 | .................................*X | 271.924 | | 263.524 | ................................**...*..**...*...**..*X | 362.700 | | 348.680 | ...........................................**X | 348.680 | | 344.680 | ...........................................X | 344.680 | | 320.696 | ........................................X | 320.696 | | 292.696 | ....................................**.**......*...........*..**..***........**....*..X | 576.857 | | 559.057 | ......................................................................*X | 559.057 | | 554.444 | .....................................................................X | 554.444 | | 544.044 | ....................................................................*X | 544.044 | | 520.644 | .................................................................*X | 520.644 | | 460.838 | .........................................................X | 460.838 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 21:46:34,825 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 47.4491144594 | 3693387776 | 2456006656 | 0 / 3909087232 | 168600 / 337367 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 61.8504772004 | 3693387776 | 2478247936 | 0 / 3909087232 | 168571 / 335852 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 51.7668300232 | 4201836544 | 2946011136 | 0 / 3758092288 | 168400 / 337600 | 7.0.0-5085-enterprise | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-02 21:46:49,076 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 21:48:08,346 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 153048424 | 257247096 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 153091192 | 273067352 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 432020744 | 451717609 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-02 21:48:08,348 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-02 21:48:15,796 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_in 2021-05-02 21:48:15,796 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_in 2021-05-02 21:48:15,894 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 21:48:22,668 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 21:48:37,075 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 10.0025387154 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 18.0182463254 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 14.6595528455 | Cluster node | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-02 21:48:37,099 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 21:48:42,273 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 21:48:47,878 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.3 2021-05-02 21:48:53,072 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 5.24 2021-05-02 21:48:58,365 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 9.54 2021-05-02 21:49:03,470 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 12.44 2021-05-02 21:49:04,029 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:jQ-21-83000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-309 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-299 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:04,029 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:jQ-21-83000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-309 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-299 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,026 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in bucket1:_default:8LwWZTytfyH-31-322000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,028 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket1:_default:_default +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,029 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in bucket1:_default:m679%vKGFP2dCg1v7qa-31-318000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,032 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in bucket1:_default:KIsDPxyRaHci-31-329000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,035 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in bucket1:_default:lLAj2Q6IS-31-326000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,036 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in bucket1:_default:Bn-31-323000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,026 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in bucket1:_default:w88tp5usO3iT30x8-31-327000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,061 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in bucket1:_default:V8hTe-TeKlRc-31-324000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:08,730 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 16.6 2021-05-02 21:49:13,903 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 19.95 2021-05-02 21:49:16,503 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:jQ-21-83000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-413 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-583 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:16,505 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:jQ-21-83000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-413 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-583 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:19,006 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 23.72 2021-05-02 21:49:19,085 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in bucket1:24R5X%-20-657000:T-20-774000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:19,085 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in bucket1:24R5X%-20-657000:T-20-774000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-565 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:24,127 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 27.48 2021-05-02 21:49:28,270 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:jQ-21-83000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-737 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:28,273 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:jQ-21-83000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-737 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:28,549 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in bucket1:24R5X%-20-657000:T-20-774000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-701 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-754 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-691 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:28,552 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in bucket1:24R5X%-20-657000:T-20-774000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-701 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-754 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-691 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:29,259 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.63 2021-05-02 21:49:30,645 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:eDgXVzrC-21-84000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-788 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-623 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-666 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:30,648 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:eDgXVzrC-21-84000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-623 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-788 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-666 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:35,546 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 21:49:38,844 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:eDgXVzrC-21-84000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-922 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:38,846 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket1:I7gY-20-944000:eDgXVzrC-21-84000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-922 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:40,696 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 38.06 2021-05-02 21:49:45,875 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 41.16 2021-05-02 21:49:48,016 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1480 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:48,016 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1480 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:48,017 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1480 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:48,019 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1480 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:48,019 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1480 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:48,023 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1480 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:48,071 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1480 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:48,072 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1480 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:51,002 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 44.49 2021-05-02 21:49:56,111 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 48.68 2021-05-02 21:49:56,556 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:56,559 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1616 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:56,562 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:56,563 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:56,565 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1616 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:56,565 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1616 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:56,585 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:56,586 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1616 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:49:57,532 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in bucket2:scope1:p1cum%ikqQQGZW-31-266000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:01,292 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 52.43 2021-05-02 21:50:04,082 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:04,085 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:04,085 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:04,085 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:04,085 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:04,086 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:04,476 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:04,477 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1873 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:06,440 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 56.04 2021-05-02 21:50:10,980 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:10,980 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:10,980 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:10,982 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:10,982 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:11,003 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:11,552 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 59.6 2021-05-02 21:50:16,684 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 63.71 2021-05-02 21:50:18,506 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:18,506 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:18,506 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:18,507 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:18,506 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:18,509 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:18,509 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:18,509 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:21,153 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in bucket2:9yGwyK_zeTjU-AT-31-286000:u_r9PO22QIjNN-22-492000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-187 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:21,154 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in bucket2:9yGwyK_zeTjU-AT-31-286000:u_r9PO22QIjNN-22-492000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-187 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:21,813 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 21:50:26,358 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2493 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2452 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,358 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2493 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2452 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,361 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2493 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2452 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,361 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2493 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2452 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,371 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2493 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2452 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,372 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:Y31FXX-19-449000:M-19-946000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2493 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2452 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,382 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2493 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2452 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,384 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2493 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2452 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:26,944 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 69.4 2021-05-02 21:50:32,119 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 73.04 2021-05-02 21:50:35,266 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:aEODihL4EJDac1b-31-391000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-915 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-932 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-950 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:37,315 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 76.45 2021-05-02 21:50:39,812 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-327 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-368 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-377 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-356 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-224 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-364 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-221 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-277 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-332 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-350 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:40,724 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-58 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-57 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-169 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-92 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-141 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:41,023 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-505 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-633 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-556 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-553 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-595 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:42,539 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 79.79 2021-05-02 21:50:45,303 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:k6PfjEu5QBsywXP-22-214000:FodQvwSyM_uWCL9ujeY-22-331000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:45,305 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:k6PfjEu5QBsywXP-22-214000:FodQvwSyM_uWCL9ujeY-22-331000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-869 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-825 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:46,957 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:mPgy1-31-389000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-629 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-636 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-505 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-556 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-553 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-642 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-595 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:47,729 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 82.42 2021-05-02 21:50:52,891 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 85.65 2021-05-02 21:50:56,009 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:_default:fZX-h3Npt5qU-31-391000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-416 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-405 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-435 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-479 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-410 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-422 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-440 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:50:58,028 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 89.4 2021-05-02 21:50:59,562 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-417 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-403 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-437 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-432 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-488 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-464 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-486 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-450 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-461 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:00,158 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:00,161 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2565 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:00,171 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2604 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:00,173 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2604 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:02,767 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-319 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-316 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-313 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-322 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-289 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-345 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-246 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-365 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-343 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-340 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-286 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-283 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:02,839 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:i-31-386000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-603 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-547 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-569 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-665 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-534 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-571 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-682 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-591 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-592 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:03,223 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 92.25 2021-05-02 21:51:08,061 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:GoM_WGcGc9%Mr-31-393000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-26 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-117 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-30 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-63 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-104 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-75 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-179 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-70 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-163 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:08,125 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:_default:GoM_WGcGc9%Mr-31-393000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-639 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-647 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-544 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-665 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-676 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-611 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-534 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-620 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-571 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-682 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:08,486 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 95.53 2021-05-02 21:51:09,611 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:k6PfjEu5QBsywXP-22-214000:FodQvwSyM_uWCL9ujeY-22-331000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1061 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:09,612 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:k6PfjEu5QBsywXP-22-214000:FodQvwSyM_uWCL9ujeY-22-331000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1061 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:10,091 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:mPgy1-31-389000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-704 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-745 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-742 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-842 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-792 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:10,232 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:_default:tucQJg-31-387000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-608 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-639 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-537 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-647 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-676 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-611 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-534 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-620 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-664 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-571 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-682 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:10,704 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-704 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-878 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-813 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-898 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-811 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-874 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-731 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-742 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-741 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-793 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-771 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:11,426 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:i-31-386000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-26 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-117 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-30 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-63 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-104 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-75 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-179 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-70 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-163 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:13,694 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 98.48 2021-05-02 21:51:15,987 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:_default:tucQJg-31-387000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-29 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-36 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-14 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-84 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-51 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-102 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-91 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-60 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:18,482 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:18,483 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2755 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:21,888 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:GoM_WGcGc9%Mr-31-393000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-249 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-206 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-305 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-243 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-353 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-321 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-396 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-295 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:22,759 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:_default:GoM_WGcGc9%Mr-31-393000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-836 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-758 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-831 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:23,203 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:k6PfjEu5QBsywXP-22-214000:FodQvwSyM_uWCL9ujeY-22-331000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:23,203 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:k6PfjEu5QBsywXP-22-214000:FodQvwSyM_uWCL9ujeY-22-331000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1379 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:23,756 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:mPgy1-31-389000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-964 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-976 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-941 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:23,877 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-02 21:51:23,959 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 166.885000229 sec 2021-05-02 21:51:24,908 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:i-31-386000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-249 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-305 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-378 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-243 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-353 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-295 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:25,020 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-964 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:25,085 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:_default:tucQJg-31-387000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-738 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-758 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-831 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:25,085 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:25,085 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:i-31-386000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-747 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-758 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-767 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-831 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-861 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-884 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:25,088 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2755 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:25,811 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in default:_default:jimPC144KX9J-31-391000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-434 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-421 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-477 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-453 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:26,542 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:_default:o%yfQ-31-384000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-617 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-577 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-687 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 21:51:34,709 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-02 21:51:34,710 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-02 21:51:34,710 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-02 21:51:47,019 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-02 21:51:47,032 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-02 21:51:47,035 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 10 docs from 0 to 5000 2021-05-02 21:51:47,039 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-02 21:51:47,055 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:51:47,055 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:51:47,058 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:51:47,059 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:51:47,059 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:51:47,061 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:51:47,063 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:51:47,065 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 2 docs from 0 to 3 2021-05-02 21:54:13,572 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-02 21:54:13,575 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-02 21:54:13,585 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 10 docs from 0 to 500 2021-05-02 21:54:13,586 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 21:54:13,589 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 29 docs from 0 to 500 2021-05-02 21:54:13,591 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-02 21:54:13,595 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 33 docs from 500 to 980 2021-05-02 21:54:13,596 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 33 docs from 0 to 500 2021-05-02 21:54:13,598 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 38 docs from 500 to 980 2021-05-02 21:54:13,601 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-02 21:54:13,602 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 4 docs from 2500 to 2900 2021-05-02 21:54:13,607 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 29 docs from 500 to 980 2021-05-02 21:54:13,608 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 24 docs from 0 to 500 2021-05-02 21:54:13,608 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 29 docs from 500 to 980 2021-05-02 21:54:13,611 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 26 docs from 0 to 500 2021-05-02 21:54:13,612 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 28 docs from 500 to 980 2021-05-02 21:54:13,614 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 14 docs from 0 to 500 2021-05-02 21:54:13,615 | test | ERROR | pool-2-thread-4 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-02 21:59:43,392 | test | INFO | pool-2-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****........*........*......*X | 55.604 | | 36.904 | .............X | 36.904 | | 36.004 | .............***......X | 52.983 | | 32.283 | ...........X | 32.283 | | 20.283 | .......X | 20.283 | | 9.707 | ..**....*..............*.....**......**X | 82.899 | | 77.010 | .............................X | 77.010 | | 70.097 | ..........................X | 70.097 | | 63.704 | ........................**.*............*........**......................................**...X | 222.021 | | 220.487 | ......................................................................................X | 220.487 | | 217.351 | .....................................................................................*X | 217.351 | | 214.076 | ...................................................................................X | 214.076 | | 204.376 | ...............................................................................X | 204.376 | | 159.250 | ..............................................................X | 159.250 | | 154.337 | ............................................................**................................**.......*X | 252.527 | | 237.860 | .............................................................................................*X | 237.860 | | 217.160 | ....................................................................................*.........X | 237.839 | | 175.300 | ....................................................................X | 175.300 | | 170.994 | ..................................................................*X | 170.994 | | 147.594 | .........................................................*...............*....*X | 195.240 | | 169.433 | ..................................................................**..**..........*X | 200.109 | | 182.391 | .......................................................................*..........**...*.........*X | 239.478 | | 209.161 | .................................................................................X | 209.161 | | 203.700 | ...............................................................................*X | 203.700 | | 196.300 | ............................................................................**..**X | 200.638 | | 196.579 | ............................................................................X | 196.579 | | 190.662 | ..........................................................................X | 190.662 | | 178.424 | .....................................................................*X | 178.424 | | 158.011 | .............................................................*X | 158.011 | | 118.844 | ..............................................**...*X | 126.715 | | 112.400 | ...........................................*......**..*......*X | 148.500 | | 127.070 | .................................................X | 127.070 | | 114.670 | ............................................*X | 114.670 | | 101.370 | .......................................*....X | 112.426 | | 107.912 | .........................................**.X | 109.312 | | 86.086 | .................................*X | 86.086 | | 78.907 | ..............................*X | 78.907 | | 8.307 | ..**X | 8.307 | | 0.600 | *X | 0.600 | | 0.400 | *****.X | 4.300 | | 4.200 | *X | 4.200 | | 2.500 | X | 2.500 | | 0.300 | *X | 0.300 | | 0.200 | *X | 0.200 | | 0.100 | ***********X | 1.300 | | 1.100 | *X | 1.100 | | 0.900 | X | 0.900 | | 0.800 | *******X | 2.099 | | 1.599 | *X | 1.599 | | 1.201 | X | 1.201 | | 1.101 | *X | 1.101 | | 0.501 | X | 0.501 | | 0.000 | **********X | 2.500 | | 2.300 | ***.**X | 2.900 | | 2.400 | X | 2.400 | | 2.200 | *X | 2.200 | | 1.800 | **.****X | 4.300 | | 2.800 | *X | 2.800 | | 2.200 | *X | 2.200 | | 1.000 | *X | 1.000 | | 0.900 | *X | 0.900 | | 0.900 | X | 0.900 | | 0.500 | *X | 0.500 | | 0.300 | *****X | 1.800 | | 1.300 | X | 1.300 | | 0.900 | X | 0.900 | | 0.800 | *****.*X | 3.300 | | 3.200 | X | 3.200 | | 2.200 | X | 2.200 | | 1.500 | *X | 1.500 | | 1.300 | *****X | 2.400 | | 1.600 | *X | 1.600 | | 1.100 | X | 1.100 | | 0.600 | X | 0.600 | | 0.300 | ********X | 1.600 | | 1.200 | X | 1.200 | | 0.500 | X | 0.500 | | 0.300 | *X | 0.300 | | 0.200 | *****X | 1.600 | | 1.200 | X | 1.200 | | 1.100 | *X | 1.100 | | 0.500 | X | 0.500 | | 0.500 | ****X | 0.600 | | 0.500 | **X | 0.800 | | 0.800 | *****.*X | 3.002 | | 2.702 | *X | 2.702 | | 2.200 | X | 2.200 | | 1.400 | ***X | 1.600 | | 1.500 | ***X | 1.600 | | 1.500 | X | 1.500 | | 1.500 | *X | 1.500 | | 1.200 | *****X | 1.700 | | 1.500 | X | 1.500 | | 1.200 | *******.*X | 4.300 | | 4.200 | ***X | 4.800 | | 3.500 | X | 3.500 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 21:59:44,546 | test | INFO | pool-2-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..**..........*.........**X | 50.502 | | 30.602 | ............X | 30.602 | | 16.602 | ......***.........*.....*.........**.*..........**..**...........X | 128.883 | | 121.583 | ..................................................X | 121.583 | | 118.977 | .................................................*X | 118.977 | | 91.403 | .....................................**..........X | 114.520 | | 112.288 | ..............................................*.......*X | 128.288 | | 99.677 | .........................................*........*..**.X | 124.530 | | 95.692 | .......................................X | 95.692 | | 95.164 | .......................................**...*...**.....**...........*............X | 174.049 | | 153.549 | ................................................................X | 153.549 | | 131.893 | .......................................................X | 131.893 | | 115.032 | ................................................**......*.*................X | 171.000 | | 158.395 | ..................................................................**.......X | 174.895 | | 146.755 | .............................................................*....X | 156.952 | | 153.152 | ................................................................*........X | 173.534 | | 162.751 | ....................................................................*X | 162.751 | | 141.746 | ...........................................................X | 141.746 | | 129.505 | ......................................................*....**...........*.............**...X | 203.776 | | 202.363 | .....................................................................................*..............*X | 234.704 | | 233.704 | ..................................................................................................*X | 233.704 | | 211.204 | ........................................................................................X | 211.204 | | 116.923 | ................................................X | 116.923 | | 83.711 | ..................................*X | 83.711 | | 76.205 | ...............................X | 76.205 | | 55.182 | ......................**..*X | 58.912 | | 50.124 | ....................*X | 50.124 | | 45.806 | ..................*X | 45.806 | | 43.000 | .................*....**..*X | 57.600 | | 49.689 | ....................*X | 49.689 | | 41.989 | ................*X | 41.989 | | 30.189 | ...........X | 30.189 | | 4.000 | *X | 4.000 | | 0.000 | ***************....*.............**.........**......*.......X | 92.918 | | 88.018 | ....................................*X | 88.018 | | 84.606 | ...................................*..***.X | 91.573 | | 91.192 | .....................................X | 91.192 | | 85.192 | ...................................**...**...*............X | 129.029 | | 128.507 | .....................................................**..X | 132.900 | | 130.088 | ......................................................***...*.*........*X | 157.587 | | 150.887 | ...............................................................*X | 150.887 | | 145.987 | .............................................................X | 145.987 | | 112.222 | ..............................................X | 112.222 | | 107.622 | ............................................*X | 107.622 | | 98.422 | ........................................X | 98.422 | | 92.300 | ......................................*X | 92.300 | | 87.700 | ....................................*X | 87.700 | | 87.400 | ....................................X | 87.400 | | 72.800 | ..............................X | 72.800 | | 65.698 | ..........................*X | 65.698 | | 50.198 | ....................*X | 50.198 | | 9.497 | ...X | 9.497 | | 2.100 | **.X | 3.700 | | 2.400 | X | 2.400 | | 2.100 | *X | 2.100 | | 0.800 | **X | 1.100 | | 0.800 | *X | 0.800 | | 0.400 | ***X | 0.400 | | 0.200 | *****..*X | 5.500 | | 5.400 | .*X | 5.400 | | 4.300 | X | 4.300 | | 1.800 | *X | 1.800 | | 1.100 | ********X | 1.700 | | 1.600 | *******.***X | 3.900 | | 3.000 | X | 3.000 | | 1.200 | X | 1.200 | | 0.200 | ******X | 1.700 | | 1.400 | X | 1.400 | | 0.700 | X | 0.700 | | 0.500 | ****X | 0.800 | | 0.500 | *X | 0.500 | | 0.200 | X | 0.200 | | 0.000 | ****.*.X | 6.301 | | 4.801 | .*X | 4.801 | | 3.101 | X | 3.101 | | 0.200 | *X | 0.200 | | 0.200 | X | 0.200 | | 0.200 | *****X | 0.800 | | 0.300 | ****.***X | 3.299 | | 2.601 | X | 2.601 | | 1.400 | *X | 1.400 | | 0.900 | *******.***X | 4.201 | | 1.300 | *X | 1.300 | | 0.000 | *********.*X | 3.300 | | 2.600 | X | 2.600 | | 2.400 | *X | 2.400 | | 1.700 | *****.X | 2.500 | | 2.100 | X | 2.100 | | 1.900 | X | 1.900 | +---------+--------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 21:59:45,364 | test | INFO | pool-2-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **********....**X | 54.386 | | 30.786 | .X | 30.786 | | 21.386 | X | 21.386 | | 12.806 | ***.**.*..*..**....*.X | 142.401 | | 139.001 | ...........*X | 139.001 | | 96.419 | .......*.......**.....*X | 224.837 | | 169.718 | ..............X | 169.718 | | 138.718 | ...........X | 138.718 | | 96.307 | .......**..*.X | 126.842 | | 111.760 | ........X | 111.760 | | 82.560 | ......X | 82.560 | | 52.624 | ...***...*...**..*.........*........*.......*X | 405.461 | | 287.863 | ........................X | 287.863 | | 182.700 | ...............X | 182.700 | | 105.490 | ........**......*........X | 265.425 | | 264.426 | ......................X | 264.426 | | 171.526 | ..............X | 171.526 | | 88.113 | ......X | 88.113 | | 62.833 | ....**..*...*...**....*.......*.**....**..............X | 482.796 | | 475.196 | .........................................*X | 475.196 | | 471.696 | .........................................X | 471.696 | | 432.082 | .....................................*.**.*........**....*X | 584.317 | | 361.803 | ...............................**X | 361.803 | | 294.630 | .........................*...*............**...........*...X | 626.600 | | 626.600 | ......................................................*.X | 637.276 | | 539.076 | ...............................................*X | 539.076 | | 457.976 | .......................................X | 457.976 | | 349.578 | ..............................X | 349.578 | | 335.226 | ............................**....*.....**......**..............................***.......X | 909.305 | | 907.305 | ................................................................................X | 907.305 | | 887.484 | ..............................................................................*X | 887.484 | | 878.584 | .............................................................................**...............*X | 1051.768 | | 1045.768 | ............................................................................................*X | 1045.768 | | 928.568 | .................................................................................X | 928.568 | | 663.332 | ..........................................................*X | 663.332 | | 575.032 | ..................................................*X | 575.032 | | 542.471 | ...............................................X | 542.471 | | 489.238 | ..........................................***.*..X | 522.249 | | 522.249 | .............................................*..**......*...*X | 646.559 | | 626.947 | .......................................................*X | 626.947 | | 575.800 | ..................................................X | 575.800 | | 461.461 | ........................................****..........**.**...*......X | 683.551 | | 683.551 | ............................................................*X | 689.351 | | 645.951 | ........................................................X | 645.951 | | 473.500 | .........................................*X | 473.500 | | 399.600 | ..................................*X | 399.600 | | 374.900 | ................................X | 374.900 | | 354.600 | ..............................**......*..........*....................................**......**.*..**......*.*.*X | 1119.513 | | 1052.575 | .............................................................................................*X | 1052.575 | | 1033.240 | ...........................................................................................X | 1033.240 | | 925.440 | .................................................................................*X | 925.440 | | 899.112 | ...............................................................................*X | 899.112 | | 696.433 | .............................................................X | 696.433 | | 696.433 | .............................................................X | 696.433 | | 618.500 | ......................................................*X | 618.500 | | 565.200 | .................................................X | 565.200 | | 500.000 | ...........................................**....**....*.....X | 647.347 | | 634.099 | .......................................................X | 634.099 | | 634.099 | .......................................................*.*......**....**.....*.............*....*X | 997.820 | | 980.720 | ......................................................................................X | 980.720 | | 951.800 | ....................................................................................*X | 951.800 | | 918.900 | .................................................................................X | 918.900 | | 918.900 | .................................................................................*.*....**X | 977.600 | | 917.431 | ................................................................................X | 917.431 | | 694.131 | .............................................................*X | 694.131 | | 642.331 | ........................................................*X | 642.331 | | 625.460 | ......................................................X | 625.460 | | 568.560 | .................................................X | 568.560 | | 556.360 | ................................................*X | 556.360 | | 551.700 | ................................................*.....**..**.*....*.**.X | 711.724 | | 702.418 | .............................................................****.*......*.*X | 786.484 | | 754.984 | ..................................................................X | 754.984 | | 617.000 | ......................................................X | 617.000 | | 588.676 | ...................................................*X | 588.676 | | 584.417 | ...................................................**X | 588.817 | | 582.363 | ...................................................*X | 582.363 | | 534.779 | ..............................................*X | 534.779 | | 531.756 | ..............................................*X | 531.756 | | 529.910 | ..............................................*.......**..*....*.....**.**.**.*X | 771.395 | | 726.495 | ...............................................................X | 726.495 | | 691.795 | ............................................................*X | 691.795 | | 657.395 | .........................................................**..*X | 672.000 | | 666.400 | ..........................................................*X | 666.400 | | 656.100 | .........................................................X | 656.100 | | 642.887 | ........................................................**X | 644.287 | | 640.887 | ........................................................**....*...**....*.........*...**.**.**X | 923.800 | | 918.800 | .................................................................................X | 918.800 | | 869.100 | ............................................................................X | 869.100 | +----------+--------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 21:59:45,434 | test | WARNING | Thread-2380 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:V8hTe-TeKlRc-31-324000 delete failed keys from task: [u'test_collections-2', u'test_collections-0'] 2021-05-02 21:59:45,434 | test | WARNING | Thread-2381 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:KIsDPxyRaHci-31-329000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:45,437 | test | WARNING | Thread-2397 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:lLAj2Q6IS-31-326000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:45,438 | test | WARNING | Thread-2399 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:_default delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:45,450 | test | WARNING | Thread-2401 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:Bn-31-323000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:45,450 | test | WARNING | Thread-2416 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:I7gY-20-944000:jQ-21-83000 create failed keys from task: [u'test_collections-737', u'test_collections-309', u'test_collections-413', u'test_collections-299', u'test_collections-583'] 2021-05-02 21:59:45,473 | test | WARNING | Thread-2404 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:8LwWZTytfyH-31-322000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:45,505 | test | WARNING | Thread-2532 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:J92-20-248000 create failed keys from task: [u'test_collections-1786', u'test_collections-1480', u'test_collections-2493', u'test_collections-2231', u'test_collections-2452', u'test_collections-1510', u'test_collections-1653', u'test_collections-1873', u'test_collections-2503', u'test_collections-1616', u'test_collections-2146'] 2021-05-02 21:59:45,528 | test | WARNING | Thread-2607 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:9yGwyK_zeTjU-AT-31-286000:u_r9PO22QIjNN-22-492000 create failed keys from task: [u'test_collections-187'] 2021-05-02 21:59:45,529 | test | WARNING | Thread-2613 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Y31FXX-19-449000:SMmWQUxEJ-19-945000 create failed keys from task: [u'test_collections-1786', u'test_collections-1480', u'test_collections-2493', u'test_collections-2231', u'test_collections-2452', u'test_collections-1510', u'test_collections-1653', u'test_collections-1873', u'test_collections-2503', u'test_collections-1616', u'test_collections-2146'] 2021-05-02 21:59:45,529 | test | WARNING | Thread-2614 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Y31FXX-19-449000:M-19-946000 create failed keys from task: [u'test_collections-1786', u'test_collections-1480', u'test_collections-2493', u'test_collections-2231', u'test_collections-2452', u'test_collections-1510', u'test_collections-1653', u'test_collections-1873', u'test_collections-2503', u'test_collections-1616', u'test_collections-2146'] 2021-05-02 21:59:45,542 | test | WARNING | Thread-2533 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:CHnzMZHeu4q1cmcu7CX-20-141000:1D-20-249000 create failed keys from task: [u'test_collections-1510', u'test_collections-1653', u'test_collections-1873', u'test_collections-1786', u'test_collections-2503', u'test_collections-1616', u'test_collections-1480', u'test_collections-2493', u'test_collections-2231', u'test_collections-2452'] 2021-05-02 21:59:45,562 | test | WARNING | Thread-2688 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:aEODihL4EJDac1b-31-391000 update failed keys from task: [u'test_collections-915', u'test_collections-977', u'test_collections-910', u'test_collections-932', u'test_collections-950'] 2021-05-02 21:59:45,563 | test | WARNING | Thread-2689 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:fZX-h3Npt5qU-31-391000 delete failed keys from task: [u'test_collections-438', u'test_collections-416', u'test_collections-405', u'test_collections-435', u'test_collections-479', u'test_collections-410', u'test_collections-422', u'test_collections-455', u'test_collections-474', u'test_collections-440'] 2021-05-02 21:59:45,568 | test | WARNING | Thread-2691 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:jimPC144KX9J-31-391000 update failed keys from task: [u'test_collections-964', u'test_collections-921', u'test_collections-567', u'test_collections-840', u'test_collections-812', u'test_collections-813', u'test_collections-898', u'test_collections-811', u'test_collections-731', u'test_collections-896', u'test_collections-771', u'test_collections-707', u'test_collections-704', u'test_collections-505', u'test_collections-868', u'test_collections-821', u'test_collections-742', u'test_collections-586', u'test_collections-741', u'test_collections-540', u'test_collections-580', u'test_collections-718', u'test_collections-878', u'test_collections-757', u'test_collections-712', u'test_collections-874', u'test_collections-633', u'test_collections-556', u'test_collections-553', u'test_collections-595', u'test_collections-793', u'test_collections-871', u'test_collections-791'] 2021-05-02 21:59:45,569 | test | WARNING | Thread-2697 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:i-31-386000 update failed keys from task: [u'test_collections-603', u'test_collections-527', u'test_collections-568', u'test_collections-843', u'test_collections-569', u'test_collections-767', u'test_collections-720', u'test_collections-688', u'test_collections-884', u'test_collections-763', u'test_collections-682', u'test_collections-534', u'test_collections-697', u'test_collections-694', u'test_collections-571', u'test_collections-702', u'test_collections-824', u'test_collections-747', u'test_collections-788', u'test_collections-547', u'test_collections-665', u'test_collections-861', u'test_collections-516', u'test_collections-758', u'test_collections-711', u'test_collections-754', u'test_collections-831', u'test_collections-591', u'test_collections-592'] 2021-05-02 21:59:45,571 | test | WARNING | Thread-2694 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:GoM_WGcGc9%Mr-31-393000 update failed keys from task: [u'test_collections-849', u'test_collections-528', u'test_collections-726', u'test_collections-647', u'test_collections-568', u'test_collections-843', u'test_collections-602', u'test_collections-720', u'test_collections-688', u'test_collections-763', u'test_collections-682', u'test_collections-654', u'test_collections-611', u'test_collections-534', u'test_collections-697', u'test_collections-694', u'test_collections-571', u'test_collections-829', u'test_collections-544', u'test_collections-665', u'test_collections-620', u'test_collections-836', u'test_collections-639', u'test_collections-516', u'test_collections-758', u'test_collections-676', u'test_collections-797', u'test_collections-831', u'test_collections-752'] 2021-05-02 21:59:45,585 | test | WARNING | Thread-2703 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:o%yfQ-31-384000 update failed keys from task: [u'test_collections-617', u'test_collections-648', u'test_collections-577', u'test_collections-687'] 2021-05-02 21:59:45,585 | test | WARNING | Thread-2704 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:tucQJg-31-387000 update failed keys from task: [u'test_collections-608', u'test_collections-726', u'test_collections-647', u'test_collections-568', u'test_collections-602', u'test_collections-688', u'test_collections-763', u'test_collections-682', u'test_collections-738', u'test_collections-537', u'test_collections-658', u'test_collections-611', u'test_collections-534', u'test_collections-697', u'test_collections-694', u'test_collections-571', u'test_collections-867', u'test_collections-665', u'test_collections-621', u'test_collections-620', u'test_collections-664', u'test_collections-639', u'test_collections-516', u'test_collections-758', u'test_collections-676', u'test_collections-797', u'test_collections-831', u'test_collections-752'] 2021-05-02 21:59:45,594 | test | WARNING | Thread-2700 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:mPgy1-31-389000 update failed keys from task: [u'test_collections-805', u'test_collections-843', u'test_collections-964', u'test_collections-921', u'test_collections-567', u'test_collections-842', u'test_collections-642', u'test_collections-614', u'test_collections-976', u'test_collections-896', u'test_collections-651', u'test_collections-629', u'test_collections-707', u'test_collections-704', u'test_collections-505', u'test_collections-824', u'test_collections-821', u'test_collections-745', u'test_collections-742', u'test_collections-864', u'test_collections-941', u'test_collections-586', u'test_collections-862', u'test_collections-781', u'test_collections-580', u'test_collections-718', u'test_collections-837', u'test_collections-636', u'test_collections-757', u'test_collections-679', u'test_collections-712', u'test_collections-556', u'test_collections-798', u'test_collections-553', u'test_collections-872', u'test_collections-595', u'test_collections-871', u'test_collections-792'] 2021-05-02 21:59:45,595 | test | WARNING | Thread-2638 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:scope1:p1cum%ikqQQGZW-31-266000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:45,611 | test | WARNING | Thread-2720 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 delete failed keys from task: [u'test_collections-403', u'test_collections-327', u'test_collections-368', u'test_collections-446', u'test_collections-488', u'test_collections-364', u'test_collections-486', u'test_collections-362', u'test_collections-417', u'test_collections-215', u'test_collections-377', u'test_collections-332', u'test_collections-450', u'test_collections-290', u'test_collections-300', u'test_collections-224', u'test_collections-221', u'test_collections-464', u'test_collections-461', u'test_collections-385', u'test_collections-437', u'test_collections-478', u'test_collections-432', u'test_collections-356', u'test_collections-232', u'test_collections-474', u'test_collections-277', u'test_collections-431', u'test_collections-350'] 2021-05-02 21:59:45,683 | test | WARNING | Thread-2406 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:m679%vKGFP2dCg1v7qa-31-318000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:45,839 | test | WARNING | Thread-2758 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:k6PfjEu5QBsywXP-22-214000:FodQvwSyM_uWCL9ujeY-22-331000 create failed keys from task: [u'test_collections-939', u'test_collections-1061', u'test_collections-869', u'test_collections-825', u'test_collections-1379'] 2021-05-02 21:59:46,015 | test | WARNING | Thread-2417 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:I7gY-20-944000:eDgXVzrC-21-84000 create failed keys from task: [u'test_collections-618', u'test_collections-922', u'test_collections-623', u'test_collections-788', u'test_collections-666'] 2021-05-02 21:59:46,017 | test | WARNING | Thread-2694 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:GoM_WGcGc9%Mr-31-393000 delete failed keys from task: [u'test_collections-249', u'test_collections-206', u'test_collections-126', u'test_collections-243', u'test_collections-321', u'test_collections-163', u'test_collections-179', u'test_collections-135', u'test_collections-295', u'test_collections-170', u'test_collections-76', u'test_collections-33', u'test_collections-35', u'test_collections-305', u'test_collections-30', u'test_collections-104', u'test_collections-75', u'test_collections-2', u'test_collections-26', u'test_collections-117', u'test_collections-63', u'test_collections-278', u'test_collections-70', u'test_collections-353', u'test_collections-396', u'test_collections-152'] 2021-05-02 21:59:46,019 | test | WARNING | Thread-2720 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:idf4-31-434000:L8vjywYYHL2IJ-31-443000 create failed keys from task: [u'test_collections-2654', u'test_collections-2822', u'test_collections-2755', u'test_collections-2565'] 2021-05-02 21:59:46,023 | test | WARNING | Thread-2704 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:tucQJg-31-387000 delete failed keys from task: [u'test_collections-36', u'test_collections-14', u'test_collections-12', u'test_collections-73', u'test_collections-51', u'test_collections-102', u'test_collections-147', u'test_collections-60', u'test_collections-29', u'test_collections-25', u'test_collections-84', u'test_collections-135', u'test_collections-91', u'test_collections-170'] 2021-05-02 21:59:46,026 | test | WARNING | Thread-2697 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:i-31-386000 delete failed keys from task: [u'test_collections-249', u'test_collections-126', u'test_collections-243', u'test_collections-163', u'test_collections-179', u'test_collections-135', u'test_collections-378', u'test_collections-295', u'test_collections-170', u'test_collections-76', u'test_collections-33', u'test_collections-35', u'test_collections-305', u'test_collections-30', u'test_collections-104', u'test_collections-75', u'test_collections-2', u'test_collections-26', u'test_collections-117', u'test_collections-63', u'test_collections-278', u'test_collections-70', u'test_collections-353', u'test_collections-152'] 2021-05-02 21:59:46,062 | test | WARNING | Thread-2691 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:jimPC144KX9J-31-391000 delete failed keys from task: [u'test_collections-18', u'test_collections-58', u'test_collections-98', u'test_collections-57', u'test_collections-203', u'test_collections-322', u'test_collections-289', u'test_collections-169', u'test_collections-246', u'test_collections-365', u'test_collections-286', u'test_collections-283', u'test_collections-42', u'test_collections-259', u'test_collections-92', u'test_collections-255', u'test_collections-453', u'test_collections-371', u'test_collections-421', u'test_collections-345', u'test_collections-343', u'test_collections-340', u'test_collections-264', u'test_collections-141', u'test_collections-319', u'test_collections-316', u'test_collections-313', u'test_collections-434', u'test_collections-477', u'test_collections-393', u'test_collections-471', u'test_collections-7', u'test_collections-8'] 2021-05-02 21:59:46,063 | test | WARNING | Thread-2456 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:24R5X%-20-657000:T-20-774000 create failed keys from task: [u'test_collections-701', u'test_collections-754', u'test_collections-565', u'test_collections-631', u'test_collections-691'] 2021-05-02 21:59:46,065 | test | WARNING | Thread-2405 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:w88tp5usO3iT30x8-31-327000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 21:59:46,155 | test | WARNING | Thread-2691 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:jimPC144KX9J-31-391000 create failed keys from task: [u'test_collections-2822', u'test_collections-2755', u'test_collections-2604'] 2021-05-02 22:00:00,102 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 22:01:48,019 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-02 22:01:48,022 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-02 22:02:04,984 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-02 22:02:04,986 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-02 22:02:21,862 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-02 22:02:21,864 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-02 22:02:59,292 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-02 22:03:16,732 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1630001068 seconds to complete 2021-05-02 22:03:16,752 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2577.875s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_1 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_2', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 2, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-02 22:03:27,039 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-02 22:03:28,017 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-02 22:03:28,631 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c831 type=client open_count=1 channel=[id: 0xfe71d3e0, 0.0.0.0/0.0.0.0:36241] timeout=300.0>) 2021-05-02 22:03:34,670 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c832 type=client open_count=1 channel=[id: 0xf0368cad, 0.0.0.0/0.0.0.0:48605] timeout=300.0>) 2021-05-02 22:03:36,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c833 type=client open_count=1 channel=[id: 0xe6ad657b, 0.0.0.0/0.0.0.0:36905] timeout=300.0>) 2021-05-02 22:03:38,755 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c834 type=client open_count=1 channel=[id: 0xff9d907c, 0.0.0.0/0.0.0.0:37671] timeout=300.0>) 2021-05-02 22:03:40,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c835 type=client open_count=1 channel=[id: 0x30f2998a, 0.0.0.0/0.0.0.0:40705] timeout=300.0>) 2021-05-02 22:03:42,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c836 type=client open_count=1 channel=[id: 0x892ad059, 0.0.0.0/0.0.0.0:43463] timeout=300.0>) 2021-05-02 22:03:44,836 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c837 type=client open_count=1 channel=[id: 0x0ca2dbe3, 0.0.0.0/0.0.0.0:39428] timeout=300.0>) 2021-05-02 22:03:46,867 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c838 type=client open_count=1 channel=[id: 0x16937038, 0.0.0.0/0.0.0.0:41921] timeout=300.0>) 2021-05-02 22:03:48,885 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5c839 type=client open_count=1 channel=[id: 0xcbd6e80f, 0.0.0.0/0.0.0.0:60481] timeout=300.0>) 2021-05-02 22:03:50,931 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c83a type=client open_count=1 channel=[id: 0x77297df3, 0.0.0.0/0.0.0.0:56891] timeout=300.0>) 2021-05-02 22:03:58,615 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c83b type=client open_count=1 channel=[id: 0x76a21b59, 0.0.0.0/0.0.0.0:43311] timeout=300.0>) 2021-05-02 22:04:00,637 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c83c type=client open_count=1 channel=[id: 0x958ce884, 0.0.0.0/0.0.0.0:56457] timeout=300.0>) 2021-05-02 22:04:02,694 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c83d type=client open_count=1 channel=[id: 0x51e0dc0e, 0.0.0.0/0.0.0.0:57133] timeout=300.0>) 2021-05-02 22:04:04,719 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c83e type=client open_count=1 channel=[id: 0x07c6076d, 0.0.0.0/0.0.0.0:59767] timeout=300.0>) 2021-05-02 22:04:06,770 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c83f type=client open_count=1 channel=[id: 0x65e6987a, 0.0.0.0/0.0.0.0:58375] timeout=300.0>) 2021-05-02 22:04:08,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c840 type=client open_count=1 channel=[id: 0xa4d3db2a, 0.0.0.0/0.0.0.0:54431] timeout=300.0>) 2021-05-02 22:04:10,851 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c841 type=client open_count=1 channel=[id: 0x6d00e60b, 0.0.0.0/0.0.0.0:49696] timeout=300.0>) 2021-05-02 22:04:12,927 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5c842 type=client open_count=1 channel=[id: 0x332c8085, 0.0.0.0/0.0.0.0:49885] timeout=300.0>) 2021-05-02 22:04:14,964 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c843 type=client open_count=1 channel=[id: 0x8d839f31, 0.0.0.0/0.0.0.0:42292] timeout=300.0>) 2021-05-02 22:04:22,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c844 type=client open_count=1 channel=[id: 0x1847cf33, 0.0.0.0/0.0.0.0:38277] timeout=300.0>) 2021-05-02 22:04:24,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c845 type=client open_count=1 channel=[id: 0x1ceed821, 0.0.0.0/0.0.0.0:52709] timeout=300.0>) 2021-05-02 22:04:26,657 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c846 type=client open_count=1 channel=[id: 0x5d31943f, 0.0.0.0/0.0.0.0:41107] timeout=300.0>) 2021-05-02 22:04:28,693 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c847 type=client open_count=1 channel=[id: 0x228c05e1, 0.0.0.0/0.0.0.0:56949] timeout=300.0>) 2021-05-02 22:04:30,739 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c848 type=client open_count=1 channel=[id: 0xdb0ff9f6, 0.0.0.0/0.0.0.0:48132] timeout=300.0>) 2021-05-02 22:04:32,763 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c849 type=client open_count=1 channel=[id: 0x527eaa98, 0.0.0.0/0.0.0.0:55816] timeout=300.0>) 2021-05-02 22:04:34,785 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c84a type=client open_count=1 channel=[id: 0x67e265e9, 0.0.0.0/0.0.0.0:55981] timeout=300.0>) 2021-05-02 22:04:36,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c84b type=client open_count=1 channel=[id: 0x20347d95, 0.0.0.0/0.0.0.0:46609] timeout=300.0>) 2021-05-02 22:04:38,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5c84c type=client open_count=1 channel=[id: 0x71439da6, 0.0.0.0/0.0.0.0:42111] timeout=300.0>) 2021-05-02 22:04:40,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c84d type=client open_count=1 channel=[id: 0x05cd2add, 0.0.0.0/0.0.0.0:43653] timeout=300.0>) 2021-05-02 22:04:48,875 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c84e type=client open_count=1 channel=[id: 0xc27694cd, 0.0.0.0/0.0.0.0:43877] timeout=300.0>) 2021-05-02 22:04:50,897 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c84f type=client open_count=1 channel=[id: 0x42a18b06, 0.0.0.0/0.0.0.0:50833] timeout=300.0>) 2021-05-02 22:04:52,934 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c850 type=client open_count=1 channel=[id: 0x1255ede0, 0.0.0.0/0.0.0.0:60568] timeout=300.0>) 2021-05-02 22:04:54,982 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c851 type=client open_count=1 channel=[id: 0xdfb09bdb, 0.0.0.0/0.0.0.0:39313] timeout=300.0>) 2021-05-02 22:04:57,028 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c852 type=client open_count=1 channel=[id: 0xe215cdc0, 0.0.0.0/0.0.0.0:46123] timeout=300.0>) 2021-05-02 22:04:59,053 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c853 type=client open_count=1 channel=[id: 0x396e116e, 0.0.0.0/0.0.0.0:47221] timeout=300.0>) 2021-05-02 22:05:01,078 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c854 type=client open_count=1 channel=[id: 0x6a514546, 0.0.0.0/0.0.0.0:56253] timeout=300.0>) 2021-05-02 22:05:03,115 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5c855 type=client open_count=1 channel=[id: 0xd135cc50, 0.0.0.0/0.0.0.0:44677] timeout=300.0>) 2021-05-02 22:05:05,135 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5c856 type=client open_count=1 channel=[id: 0x4c9c200b, 0.0.0.0/0.0.0.0:50283] timeout=300.0>) 2021-05-02 22:05:10,444 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5c857 type=client open_count=1 channel=[id: 0x8087c3fc, 0.0.0.0/0.0.0.0:55180] timeout=300.0>) 2021-05-02 22:05:12,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5c858 type=client open_count=1 channel=[id: 0x0468b854, 0.0.0.0/0.0.0.0:52909] timeout=300.0>) 2021-05-02 22:05:14,500 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5c859 type=client open_count=1 channel=[id: 0x9fae48df, 0.0.0.0/0.0.0.0:46335] timeout=300.0>) 2021-05-02 22:05:16,526 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5c85a type=client open_count=1 channel=[id: 0x44840c4f, 0.0.0.0/0.0.0.0:45159] timeout=300.0>) 2021-05-02 22:05:18,552 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5c85b type=client open_count=1 channel=[id: 0x7be29876, 0.0.0.0/0.0.0.0:59457] timeout=300.0>) 2021-05-02 22:05:20,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 22:05:37,582 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-02 22:05:52,039 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-02 22:05:52,066 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-02 22:05:52,121 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-02 22:06:32,655 | test | INFO | pool-4-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.55284961085 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-02 22:06:32,693 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 22:06:42,793 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-02 22:06:42,825 | test | INFO | pool-4-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1700000763 sec 2021-05-02 22:06:43,226 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-02 22:07:22,005 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-02 22:07:22,325 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 526737736 | 185988120 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 504067360 | 133050532 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 555358856 | 141031895 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-02 22:07:22,325 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-02 22:08:17,648 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 22:08:17,664 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 22:19:50,829 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **................**..............*.......................*...........................................**....*X | 1530.364 | | 1157.000 | ..........................................................................**X | 1157.000 | | 1105.913 | .......................................................................X | 1105.913 | | 1081.013 | .....................................................................*.....**.*X | 1178.197 | | 1169.697 | ...........................................................................*X | 1169.697 | | 1120.800 | ........................................................................*X | 1120.800 | | 1051.843 | ...................................................................X | 1051.843 | | 865.743 | .......................................................*X | 865.743 | | 845.643 | ......................................................*X | 845.643 | | 732.327 | ..............................................*X | 732.327 | | 675.968 | ...........................................*X | 675.968 | | 654.641 | .........................................X | 654.641 | | 583.641 | .....................................X | 583.641 | | 575.978 | ....................................*X | 575.978 | | 571.605 | ....................................X | 571.605 | | 552.505 | ...................................**.*..**...........*..**..**X | 833.251 | | 830.551 | .....................................................X | 830.551 | | 825.100 | ....................................................X | 825.100 | | 776.660 | .................................................X | 776.660 | | 756.460 | ................................................*X | 756.460 | | 728.960 | ..............................................X | 728.960 | | 696.900 | ............................................*X | 696.900 | | 677.400 | ...........................................*X | 677.400 | | 597.100 | ......................................**X | 597.100 | | 596.800 | .....................................*.*.**.*...**...........*....*X | 906.958 | | 896.600 | .........................................................X | 896.600 | | 835.200 | .....................................................*X | 835.200 | | 810.800 | ...................................................X | 810.800 | | 788.500 | ..................................................X | 788.500 | | 696.000 | ............................................*X | 696.000 | | 661.100 | ..........................................**....***..X | 756.428 | | 747.128 | ...............................................X | 747.128 | | 680.743 | ...........................................*X | 680.743 | | 659.043 | ..........................................*X | 659.043 | | 616.175 | .......................................****.......**..**..*.....**X | 861.037 | | 857.537 | .......................................................X | 857.537 | | 813.018 | ....................................................X | 813.018 | | 808.918 | ...................................................*X | 808.918 | | 766.218 | .................................................X | 766.218 | | 677.600 | ...........................................*X | 677.600 | | 642.800 | .........................................*X | 642.800 | | 620.263 | .......................................*X | 620.263 | | 610.963 | ......................................*X | 610.963 | | 605.183 | ......................................X | 605.183 | | 561.055 | ...................................X | 561.055 | | 560.055 | ...................................**..*.....**..**......***.*X | 804.497 | | 757.122 | ................................................*X | 757.122 | | 742.587 | ...............................................*X | 742.587 | | 694.361 | ............................................*X | 694.361 | | 669.532 | ..........................................*X | 669.532 | | 638.932 | ........................................X | 638.932 | | 582.860 | .....................................***..**.X | 638.135 | | 637.365 | ........................................**.*......*..*X | 773.600 | | 773.187 | .................................................*.***..*...*X | 862.473 | | 853.021 | ......................................................*X | 853.021 | | 688.401 | ...........................................*X | 688.401 | | 663.636 | ..........................................*X | 663.636 | | 650.436 | .........................................X | 650.436 | | 631.527 | ........................................*X | 631.527 | | 620.827 | .......................................**.**.*X | 647.840 | | 638.240 | ........................................*.****...*......*..*X | 811.658 | | 798.158 | ...................................................X | 798.158 | | 760.419 | ................................................*X | 760.419 | | 737.870 | ...............................................*X | 737.870 | | 616.573 | .......................................*X | 616.573 | | 597.803 | ......................................*X | 597.803 | | 594.103 | .....................................X | 594.103 | | 589.095 | .....................................*X | 589.095 | | 572.695 | ....................................**..**..*X | 630.261 | | 613.061 | .......................................**.**.*......**.**X | 764.265 | | 695.164 | ............................................X | 695.164 | | 671.564 | ..........................................*X | 671.564 | | 664.264 | ..........................................*..*.*X | 711.985 | | 703.962 | ............................................X | 703.962 | | 651.120 | .........................................*X | 651.120 | | 613.820 | .......................................*X | 613.820 | | 556.914 | ...................................*X | 556.914 | | 546.814 | ..................................***..**.*..*........**...**.X | 801.013 | | 780.174 | .................................................X | 780.174 | | 767.261 | .................................................*X | 767.261 | | 740.661 | ...............................................X | 740.661 | | 738.300 | ...............................................*X | 738.300 | | 737.900 | ...............................................*X | 737.900 | | 732.700 | ..............................................X | 732.700 | | 664.745 | ..........................................X | 664.745 | | 637.545 | ........................................*X | 637.545 | | 619.445 | .......................................X | 619.445 | | 563.144 | ...................................****.*.X | 594.659 | | 584.959 | .....................................**.*....**...**.*X | 720.476 | | 716.876 | .............................................*X | 716.876 | | 696.300 | ............................................X | 696.300 | | 684.800 | ...........................................*.**X | 692.600 | | 624.600 | .......................................*X | 624.600 | | 573.900 | ....................................***.**..*X | 624.529 | | 620.829 | .......................................*X | 620.829 | | 604.550 | ......................................*X | 604.550 | | 592.050 | .....................................X | 592.050 | | 567.214 | ....................................***..*......***.*..**..*X | 774.720 | | 758.520 | ................................................X | 758.520 | | 726.729 | ..............................................X | 726.729 | | 698.029 | ............................................**X | 698.029 | | 689.229 | ............................................X | 689.229 | | 671.271 | ..........................................*..*X | 690.071 | +----------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 22:19:51,042 | test | INFO | pool-4-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *.......*.........................................................**.............................**.......X | 47.509 | | 20.309 | .........................................X | 20.309 | | 6.709 | .............*X | 6.709 | | 0.000 | **********....**........**..............**X | 12.797 | | 10.397 | ....................X | 10.397 | | 8.001 | ...............X | 8.001 | | 3.001 | .....*X | 3.001 | | 1.801 | ..*X | 1.801 | | 1.200 | .X | 1.200 | | 0.000 | ******************************.*..X | 1.500 | | 1.499 | ..****X | 1.501 | | 1.101 | .*X | 1.101 | | 0.800 | X | 0.800 | | 0.000 | ***.*..*X | 1.500 | | 1.100 | .*X | 1.100 | | 0.800 | X | 0.800 | | 0.000 | ********..*...**X | 2.599 | | 0.400 | **X | 0.400 | | 0.000 | ********************.**...**...*X | 3.701 | | 2.501 | ....*X | 2.501 | | 1.600 | ..X | 1.600 | | 0.000 | ***.**..X | 1.500 | | 1.100 | .*X | 1.100 | | 0.800 | X | 0.800 | | 0.000 | ********..*...***X | 2.600 | | 0.400 | *X | 0.400 | | 0.000 | *************...**..*.........*X | 6.800 | | 5.600 | ..........*X | 5.600 | | 4.700 | ........X | 4.700 | | 0.700 | *X | 0.700 | | 0.300 | *.*...**..*X | 2.999 | | 2.399 | ....X | 2.399 | | 0.800 | **....*...*..........*X | 8.999 | | 6.600 | ............*X | 6.600 | | 5.700 | ..........X | 5.700 | | 3.300 | .....*....*X | 4.900 | | 4.300 | ........X | 4.300 | | 2.700 | ....*X | 2.700 | | 1.100 | .*X | 1.100 | | 0.000 | *****..*.*X | 1.500 | | 1.200 | .X | 1.200 | | 0.400 | ****...*.......*.*X | 5.600 | | 4.400 | ........X | 4.400 | | 1.200 | .*X | 1.200 | | 0.800 | **.....*X | 3.000 | | 2.200 | ...**......*.................**..........*X | 18.004 | | 4.800 | .........*X | 4.800 | | 0.000 | **.*...**...*X | 3.401 | | 3.101 | .....X | 3.101 | | 2.301 | ...X | 2.301 | | 2.300 | ...**.....*............**.*X | 10.800 | | 8.400 | ................X | 8.400 | | 2.000 | ...X | 2.000 | | 1.200 | .**..*.....**..**.......*........................*....*X | 22.094 | | 18.993 | ......................................X | 18.993 | | 12.302 | ........................**..........**...........*.......................X | 33.203 | | 29.603 | .............................................................*X | 29.603 | | 27.200 | ........................................................X | 27.200 | | 21.300 | ...........................................***X | 21.300 | | 20.700 | ..........................................X | 20.700 | | 10.600 | .....................*X | 10.600 | | 5.400 | ..........*X | 5.400 | | 1.800 | ..X | 1.800 | | 0.000 | *.....**.....*..............**.....**...*......X | 18.406 | | 16.400 | .................................*X | 16.400 | | 15.200 | ..............................X | 15.200 | | 12.000 | ........................*X | 12.000 | | 8.800 | .................X | 8.800 | | 6.400 | ............X | 6.400 | | 0.000 | ************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------+--------+ 2021-05-02 22:19:52,226 | test | INFO | pool-4-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *...*......................................................*............................**..............X | 88.820 | | 40.020 | ............................................*X | 40.020 | | 15.620 | ................*X | 15.620 | | 0.000 | *********....*.....................**.............*.X | 34.696 | | 20.696 | ......................*X | 20.696 | | 9.096 | .........*X | 9.096 | | 4.600 | ....X | 4.600 | | 0.000 | *****..**X | 2.300 | | 2.299 | .X | 2.299 | | 0.699 | *X | 0.699 | | 0.300 | *X | 0.300 | | 0.000 | *********..*.*X | 3.000 | | 2.400 | .X | 2.400 | | 0.800 | *X | 0.800 | | 0.000 | ******..*...**.*X | 5.598 | | 5.302 | ....**..****X | 7.100 | | 6.300 | ......*X | 6.300 | | 4.900 | ....**X | 4.900 | | 3.300 | ..X | 3.300 | | 2.400 | .X | 2.400 | | 0.000 | **.**..*....X | 6.701 | | 6.301 | ......*X | 6.301 | | 4.801 | ....*X | 4.801 | | 0.800 | X | 0.800 | | 0.000 | ******.***X | 1.500 | | 1.200 | X | 1.200 | | 0.400 | *X | 0.400 | | 0.000 | ************..*.**.X | 4.101 | | 4.101 | ...X | 4.101 | | 2.501 | .*X | 2.501 | | 1.600 | X | 1.600 | | 0.000 | ******.**.**...X | 4.500 | | 3.700 | ...***.*.*X | 5.600 | | 4.400 | ...X | 4.400 | | 1.199 | *X | 1.199 | | 0.800 | **.*..X | 3.000 | | 2.600 | .*X | 2.600 | | 2.300 | .X | 2.300 | | 1.500 | *X | 1.500 | | 1.100 | *X | 1.100 | | 0.000 | *******.**..***X | 3.100 | | 2.800 | ..X | 2.800 | | 1.100 | *****..X | 2.999 | | 2.199 | .*X | 2.199 | | 1.599 | X | 1.599 | | 0.000 | **.**.......*X | 7.899 | | 7.100 | ......*X | 7.100 | | 6.200 | .....X | 6.200 | | 2.200 | .*X | 2.200 | | 1.800 | .X | 1.800 | | 1.200 | X | 1.200 | | 0.400 | *X | 0.400 | | 0.000 | ***.****..**.*X | 3.900 | | 1.800 | .**X | 1.800 | | 1.000 | **....**.*X | 5.600 | | 4.400 | ...X | 4.400 | | 1.200 | X | 1.200 | | 0.000 | ***.**.**..***X | 4.101 | | 3.200 | ..X | 3.200 | | 0.800 | *X | 0.800 | | 0.000 | **..**.*X | 3.001 | | 2.401 | .X | 2.401 | | 0.801 | *X | 0.801 | | 0.400 | *.*...***X | 4.100 | | 3.200 | ..X | 3.200 | | 0.800 | **..**..*......X | 9.001 | | 7.001 | ......*X | 7.001 | | 6.701 | ......X | 6.701 | | 5.898 | .....**.*X | 7.098 | | 5.598 | .....X | 5.598 | | 2.400 | .**.*..*............**..**.X | 17.800 | | 10.600 | ..........*X | 10.600 | | 9.000 | .........X | 9.000 | | 6.900 | ......X | 6.900 | | 2.900 | ..*X | 2.900 | | 0.900 | *X | 0.900 | | 0.000 | ***..*...*......*X | 10.097 | | 8.902 | .........*X | 8.902 | | 7.402 | .......X | 7.402 | | 3.402 | ..X | 3.402 | | 3.000 | ..****X | 3.000 | | 2.200 | .*X | 2.200 | | 1.600 | X | 1.600 | | 0.000 | ************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------+--------+ 2021-05-02 22:19:52,686 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 45.2312516331 | 3693387776 | 2595635200 | 0 / 3909087232 | 101958 / 207100 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 62.2251655629 | 3693387776 | 2654109696 | 0 / 3909087232 | 101009 / 198643 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 48.4840632288 | 6078930944 | 4780810240 | 0 / 3758092288 | 100702 / 196338 | 7.0.0-5085-enterprise | | 172.23.120.206 | kv | 27.5574112735 | 3691520000 | 2740416512 | 0 / 0 | 98154 / 199347 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 52.26039783 | 4201836544 | 3132661760 | 0 / 3758092288 | 100054 / 202587 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-02 22:20:06,273 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 22:21:59,522 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 160017976 | 240055316 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 157821816 | 271861446 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 437127656 | 450977642 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-02 22:21:59,525 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-02 22:22:10,398 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_out 2021-05-02 22:22:10,398 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_out 2021-05-02 22:22:10,509 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 22:22:10,885 | test | INFO | pool-4-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 11.1644354635 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 13.4498480243 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 8.4071916941 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 6.48498612163 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.13197969543 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-02 22:22:10,931 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 22:22:17,609 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 22:22:19,404 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 22:22:24,598 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 3.24 2021-05-02 22:22:29,726 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 6.31 2021-05-02 22:22:30,607 | test | INFO | pool-4-thread-25 | [table_view:display:72] Keys failed in bucket1:scope1:STqVKrW-43-300000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:34,417 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:wh-z7L-59u2XKK6u-43-303000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:34,859 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 10.08 2021-05-02 22:22:38,888 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-118 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:38,888 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-118 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:38,891 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-118 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:38,891 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-118 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-97 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:38,891 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-118 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-97 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:38,898 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-118 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-97 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:39,992 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 12.61 2021-05-02 22:22:45,122 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 15.39 2021-05-02 22:22:50,243 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 17.63 2021-05-02 22:22:52,653 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-359 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-337 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:52,653 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-359 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-337 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:52,654 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-359 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-337 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:52,653 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-359 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:52,655 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-359 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:52,655 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-359 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:22:55,401 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 19.91 2021-05-02 22:23:00,515 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 22.43 2021-05-02 22:23:01,213 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:01,214 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-489 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:01,223 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:01,223 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-489 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:01,226 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:01,227 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-489 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:05,638 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 24.82 2021-05-02 22:23:10,142 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:10,144 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:10,150 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:10,151 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:10,154 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:10,154 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:10,753 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 27.41 2021-05-02 22:23:15,944 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 29.67 2021-05-02 22:23:21,078 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 32.14 2021-05-02 22:23:26,190 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 22:23:31,288 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 22:23:37,851 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.64 2021-05-02 22:23:43,056 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 38.15 2021-05-02 22:23:45,144 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in bucket2:_default:DeexDdTvq5d8ULw-43-274000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:46,654 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in bucket2:1E8mx7ihnG5VKwm-14-392000:sb1myTZTpjAemJFo-14-513000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:46,655 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in bucket2:1E8mx7ihnG5VKwm-14-392000:sb1myTZTpjAemJFo-14-513000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2291 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2379 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2244 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2301 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:47,825 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:eVg_RQ6j3l1KfWdt-15-138000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-324 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-314 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-284 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:47,825 | test | INFO | pool-4-thread-28 | [table_view:display:72] Keys failed in bucket2:PxR8vcl4-15-336000:VyNoVs0tAwD-15-484000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-324 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-314 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-284 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:47,826 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:eVg_RQ6j3l1KfWdt-15-138000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-324 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-314 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-265 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-220 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-284 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:47,828 | test | INFO | pool-4-thread-28 | [table_view:display:72] Keys failed in bucket2:PxR8vcl4-15-336000:VyNoVs0tAwD-15-484000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-324 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-314 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-265 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-220 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-284 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:48,180 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 41.77 2021-05-02 22:23:53,292 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 44.68 2021-05-02 22:23:57,378 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:3RumrAMZUS-15-137000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1509 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1499 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:57,381 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:3RumrAMZUS-15-137000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1509 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1499 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:23:58,437 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 46.88 2021-05-02 22:24:00,367 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in bucket2:1E8mx7ihnG5VKwm-14-392000:sb1myTZTpjAemJFo-14-513000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2452 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:00,368 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in bucket2:1E8mx7ihnG5VKwm-14-392000:sb1myTZTpjAemJFo-14-513000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2535 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2452 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:00,415 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:eVg_RQ6j3l1KfWdt-15-138000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-556 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:00,417 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:eVg_RQ6j3l1KfWdt-15-138000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-556 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-431 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:00,424 | test | INFO | pool-4-thread-28 | [table_view:display:72] Keys failed in bucket2:PxR8vcl4-15-336000:VyNoVs0tAwD-15-484000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-556 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:00,424 | test | INFO | pool-4-thread-28 | [table_view:display:72] Keys failed in bucket2:PxR8vcl4-15-336000:VyNoVs0tAwD-15-484000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-556 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-431 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:03,565 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 48.97 2021-05-02 22:24:08,717 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 51.02 2021-05-02 22:24:13,924 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 52.69 2021-05-02 22:24:19,109 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 54.76 2021-05-02 22:24:24,269 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 57.14 2021-05-02 22:24:29,433 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 59.36 2021-05-02 22:24:34,529 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 61.55 2021-05-02 22:24:39,638 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 63.9 2021-05-02 22:24:42,411 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in bucket2:1E8mx7ihnG5VKwm-14-392000:sb1myTZTpjAemJFo-14-513000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2969 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:42,413 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in bucket2:1E8mx7ihnG5VKwm-14-392000:sb1myTZTpjAemJFo-14-513000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2969 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:44,755 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 65.75 2021-05-02 22:24:48,967 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:3RumrAMZUS-15-137000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:48,967 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in bucket2:23ojEOJEMmWXsPI-15-12000:3RumrAMZUS-15-137000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:49,888 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 22:24:54,996 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 22:24:55,533 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in bucket2:PxR8vcl4-15-336000:0YRwoRy-15-483000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:24:55,535 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in bucket2:PxR8vcl4-15-336000:0YRwoRy-15-483000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:00,546 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 67.63 2021-05-02 22:25:02,898 | test | INFO | pool-4-thread-25 | [table_view:display:72] Keys failed in default:XdvxYNFcKLcb-43-370000:T9PJd52Io6%zffMaJlH-43-386000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-138 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:05,693 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:1w0l-43-361000:Wbw0XG_SqFxrkgIw-%D-43-366000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-32 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-138 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:05,786 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 71.12 2021-05-02 22:25:10,045 | test | INFO | pool-4-thread-10 | [table_view:display:72] Keys failed in default:_default:BS-V0eDy_eBRyPucfm%-43-339000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-466 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-430 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:10,046 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in default:1w0l-43-361000:5zEuoY3818-43-364000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-466 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-430 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:11,046 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 74.37 2021-05-02 22:25:16,181 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 76.99 2021-05-02 22:25:18,171 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in default:XdvxYNFcKLcb-43-370000:T9PJd52Io6%zffMaJlH-43-386000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-836 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-743 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-710 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-873 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:21,289 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.0 2021-05-02 22:25:24,480 | test | INFO | pool-4-thread-22 | [table_view:display:72] Keys failed in default:1w0l-43-361000:Wbw0XG_SqFxrkgIw-%D-43-366000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-836 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-710 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-873 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:26,417 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 81.19 2021-05-02 22:25:31,456 | test | INFO | pool-4-thread-9 | [table_view:display:72] Keys failed in default:scope2:S-43-358000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-969 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:31,621 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 83.23 2021-05-02 22:25:32,862 | test | INFO | pool-4-thread-25 | [table_view:display:72] Keys failed in default:XdvxYNFcKLcb-43-370000:T9PJd52Io6%zffMaJlH-43-386000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-307 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-311 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-342 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-297 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-281 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:32,864 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:1w0l-43-361000:Wbw0XG_SqFxrkgIw-%D-43-366000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-311 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-281 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:32,921 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in default:XdvxYNFcKLcb-43-370000:T9PJd52Io6%zffMaJlH-43-386000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-927 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-916 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:32,930 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in default:1w0l-43-361000:bAvyhhv-43-367000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-601 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-612 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-644 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-550 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:32,931 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:_default:be5vh9-43-339000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-601 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-644 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-680 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:32,937 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in default:XdvxYNFcKLcb-43-370000:Pu5DCmjqYz-43-372000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-659 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-601 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-696 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-593 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-680 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:35,548 | test | INFO | pool-4-thread-22 | [table_view:display:72] Keys failed in default:1w0l-43-361000:Wbw0XG_SqFxrkgIw-%D-43-366000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-927 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-916 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:36,723 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 85.34 2021-05-02 22:25:39,141 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in default:_default:4xO2GcHI-43-341000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:39,141 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in default:_default:4xO2GcHI-43-341000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:39,141 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:_default:ARMTX8%7mdjjlzXR-43-336000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:39,142 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:_default:ARMTX8%7mdjjlzXR-43-336000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:39,296 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in default:_default:4xO2GcHI-43-341000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-63 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-154 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:41,834 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 87.55 2021-05-02 22:25:46,279 | test | INFO | pool-4-thread-25 | [table_view:display:72] Keys failed in default:XdvxYNFcKLcb-43-370000:T9PJd52Io6%zffMaJlH-43-386000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-419 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-448 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-459 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-491 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:46,279 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:1w0l-43-361000:Wbw0XG_SqFxrkgIw-%D-43-366000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-419 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-448 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-459 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-491 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:46,957 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 89.52 2021-05-02 22:25:48,023 | test | INFO | pool-4-thread-9 | [table_view:display:72] Keys failed in default:scope2:urjiC4%u8%J-43-349000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-546 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-555 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-501 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-589 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-663 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:49,141 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:_default:be5vh9-43-339000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-809 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-881 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-890 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:49,144 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in default:1w0l-43-361000:bAvyhhv-43-367000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-809 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-881 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-890 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:52,154 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 91.55 2021-05-02 22:25:56,051 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in default:_default:4xO2GcHI-43-341000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-369 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-234 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-378 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:56,697 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in default:_default:4xO2GcHI-43-341000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2867 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:56,697 | test | INFO | pool-4-thread-3 | [table_view:display:72] Keys failed in default:_default:4xO2GcHI-43-341000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2867 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:25:57,279 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 93.66 2021-05-02 22:26:00,585 | test | INFO | pool-4-thread-22 | [table_view:display:72] Keys failed in default:scope2:jWgzINZCH-h32%MLjcX-43-355000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-39 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-158 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-131 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:02,418 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 95.55 2021-05-02 22:26:04,160 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:1w0l-43-361000:eIfxvbYI%3llNeO8qL6H-43-368000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-39 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:04,398 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in default:XdvxYNFcKLcb-43-370000:Pu5DCmjqYz-43-372000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-735 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:05,088 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:_default:ARMTX8%7mdjjlzXR-43-336000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2730 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:05,089 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:_default:ARMTX8%7mdjjlzXR-43-336000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2730 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:07,549 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 97.83 2021-05-02 22:26:07,674 | test | INFO | pool-4-thread-17 | [table_view:display:72] Keys failed in default:oIWtfhL_6M25FO-17-187000:P8RJoFhf-17-319000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3165 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:07,676 | test | INFO | pool-4-thread-17 | [table_view:display:72] Keys failed in default:oIWtfhL_6M25FO-17-187000:P8RJoFhf-17-319000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3165 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:09,578 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in default:1w0l-43-361000:bAvyhhv-43-367000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-913 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-933 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:10,259 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:_default:be5vh9-43-339000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-965 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:10,717 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in default:scope2:944S4YEtlOi-43-350000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-503 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-564 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-521 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:12,694 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 99.14 2021-05-02 22:26:17,825 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-02 22:26:20,740 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in default:_default:f2MrbSa7Q-43-336000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-128 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 22:26:22,918 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-02 22:26:29,115 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-02 22:26:29,118 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 1 docs from 3 to 5 2021-05-02 22:26:29,118 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 8 docs from 0 to 5000 2021-05-02 22:26:29,118 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 8 docs from 0 to 5000 2021-05-02 22:26:29,121 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 8 docs from 0 to 5000 2021-05-02 22:26:29,125 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 22:26:33,076 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb8bb9 type=client open_count=1 channel=[id: 0x87ffe545, 0.0.0.0/0.0.0.0:57098] timeout=300.0>) 2021-05-02 22:26:33,135 | test | ERROR | pool-4-thread-18 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb8bc2 type=client open_count=1 channel=[id: 0x962b7a5c, 0.0.0.0/0.0.0.0:56692] timeout=300.0>) 2021-05-02 22:26:35,164 | test | ERROR | pool-4-thread-18 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 22:26:37,342 | test | INFO | pool-4-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 266.457000017 sec 2021-05-02 22:29:58,875 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-02 22:29:58,877 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-02 22:29:58,878 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 3 docs from 0 to 5000 2021-05-02 22:29:58,885 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-02 22:29:58,887 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-02 22:31:30,880 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 22:31:30,891 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-02 22:31:30,894 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-02 22:31:30,897 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-02 22:31:30,898 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 13 docs from 500 to 980 2021-05-02 22:31:30,898 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-02 22:31:30,900 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-02 22:31:30,901 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-02 22:31:30,901 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-02 22:31:30,903 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 12 docs from 500 to 980 2021-05-02 22:31:30,904 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 16 docs from 500 to 980 2021-05-02 22:31:30,905 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 22:31:30,907 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 5 docs from 0 to 500 2021-05-02 22:31:30,907 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 22:31:30,908 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-02 22:31:30,911 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-02 22:31:30,913 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 22:31:30,914 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-02 22:31:30,915 | test | ERROR | pool-4-thread-14 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 22:39:32,131 | test | INFO | pool-4-thread-1 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................................**..............*....*X | 536.300 | | 527.325 | ....................................................X | 527.325 | | 490.425 | .................................................*....*X | 530.574 | | 484.049 | ................................................X | 484.049 | | 392.949 | .......................................X | 392.949 | | 314.600 | ...............................*X | 314.600 | | 307.857 | ..............................*..............*......................................*.................*X | 976.696 | | 878.697 | ........................................................................................X | 878.697 | | 564.341 | ........................................................X | 564.341 | | 377.002 | .....................................*X | 377.002 | | 327.962 | ................................X | 327.962 | | 212.128 | ....................*....**.......*..........................**.......X | 639.246 | | 632.246 | ...............................................................X | 632.246 | | 513.071 | ...................................................*X | 513.071 | | 425.121 | ..........................................X | 425.121 | | 388.721 | ......................................X | 388.721 | | 317.521 | ...............................*...*X | 346.200 | | 338.600 | .................................*.**.*....*.......X | 464.038 | | 443.373 | ............................................*X | 443.373 | | 194.855 | ..................*X | 194.855 | | 154.946 | ..............**........X | 225.045 | | 225.045 | ......................*.....X | 279.720 | | 279.280 | ...........................**X | 280.980 | | 239.380 | .......................*.*.........**......X | 394.600 | | 388.400 | ......................................X | 388.400 | | 306.500 | ..............................X | 306.500 | | 235.100 | .......................***.....*.....*X | 336.085 | | 304.385 | ..............................X | 304.385 | | 135.884 | ............X | 135.884 | | 68.601 | ......*X | 68.601 | | 65.802 | .....**.........*....*.......*X | 263.415 | | 252.550 | ........................*......X | 307.550 | | 300.917 | .............................*..*X | 317.521 | | 256.687 | .........................X | 256.687 | | 219.846 | .....................X | 219.846 | | 155.554 | ..............*X | 155.554 | | 110.655 | ..........**....**......*......*....*X | 305.062 | | 298.462 | .............................X | 298.462 | | 264.975 | ..........................*X | 264.975 | | 220.453 | .....................*X | 220.453 | | 197.941 | ...................*X | 197.941 | | 191.141 | ..................**..X | 206.005 | | 175.536 | ................*X | 175.536 | | 162.136 | ...............*.*..**...*X | 217.700 | | 164.000 | ...............X | 164.000 | | 135.200 | ............*X | 135.200 | | 84.300 | .......*X | 84.300 | | 56.600 | ....****......*...**....X | 178.300 | | 140.500 | .............X | 140.500 | | 117.882 | ...........*X | 117.882 | | 64.782 | .....*X | 64.782 | | 59.882 | .....X | 59.882 | | 54.114 | ....**...**..*..**..**..*..X | 178.800 | | 173.900 | ................*X | 173.900 | | 160.500 | ...............X | 160.500 | | 144.800 | .............**....**..*.....*X | 248.600 | | 207.700 | ....................*X | 207.700 | | 179.200 | .................X | 179.200 | | 121.575 | ...........**.*..*.....**...*X | 228.387 | | 200.987 | ...................X | 200.987 | | 164.124 | ...............*X | 164.124 | | 157.178 | ...............*.....*.....****...*......*......**X | 404.470 | | 386.865 | ......................................X | 386.865 | | 339.965 | .................................X | 339.965 | | 317.209 | ...............................*X | 317.209 | | 299.991 | .............................*..*.**.....*.*X | 383.712 | | 356.511 | ...................................*X | 356.511 | | 300.900 | .............................X | 300.900 | | 258.300 | .........................X | 258.300 | | 244.741 | ........................*X | 244.741 | | 221.153 | .....................*X | 221.153 | | 212.453 | ....................*..**........*.....X | 357.636 | | 357.636 | ...................................X | 357.636 | | 355.806 | ...................................*X | 355.806 | | 286.844 | ............................X | 286.844 | | 222.044 | .....................X | 222.044 | | 213.044 | ....................**..*...*X | 259.800 | | 230.100 | ......................**.X | 242.500 | | 219.280 | .....................*..***.*X | 244.421 | | 228.921 | ......................*X | 228.921 | | 195.400 | ...................***.*.*.*X | 228.352 | | 201.231 | ...................*X | 201.231 | | 170.331 | ................X | 170.331 | | 120.231 | ...........**.**....*....*X | 210.386 | | 195.686 | ...................*.*.**...*X | 247.038 | | 198.552 | ...................X | 198.552 | | 154.735 | ..............*X | 154.735 | | 125.235 | ...........**..*.**..**...*X | 195.900 | | 180.600 | .................**X | 183.500 | | 177.900 | .................*.**X | 193.900 | | 170.700 | ................X | 170.700 | | 143.686 | .............X | 143.686 | | 128.886 | ............**.*...*..***.*X | 201.466 | | 194.231 | ..................*X | 194.231 | | 179.831 | .................X | 179.831 | | 161.666 | ...............X | 161.666 | | 153.635 | ..............**.*..*.**X | 189.963 | | 183.175 | .................**..*X | 200.902 | | 197.728 | ...................**X | 197.728 | | 179.213 | .................X | 179.213 | | 168.700 | ................**X | 170.300 | | 169.000 | ................*X | 169.000 | | 156.900 | ...............**X | 156.900 | | 155.600 | ..............*.*.****.X | 175.924 | | 171.124 | ................**X | 173.379 | | 159.379 | ...............*X | 159.379 | | 140.579 | .............*.**..*.....*X | 223.500 | | 222.500 | .....................*X | 222.500 | | 171.100 | ................**.**...**..*.**....*....**......**....*...**X | 445.718 | | 414.018 | .........................................*X | 414.018 | | 382.095 | ......................................*X | 382.095 | | 368.805 | ....................................X | 368.805 | | 354.858 | ...................................**X | 361.228 | | 321.307 | ...............................*X | 321.307 | | 266.682 | ..........................**.**..X | 293.886 | | 292.078 | ............................**..*X | 303.221 | | 300.579 | .............................*..*..**....*....**...***.**X | 456.772 | | 455.272 | .............................................*X | 455.272 | | 412.500 | .........................................X | 412.500 | | 368.800 | ....................................*X | 368.800 | | 317.300 | ...............................*****..*X | 332.146 | | 329.646 | ................................*X | 329.646 | | 326.148 | ................................X | 326.148 | | 308.347 | ..............................*X | 308.347 | | 282.763 | ...........................*X | 282.763 | | 274.101 | ...........................*..*.....*X | 345.372 | | 340.379 | .................................X | 340.379 | | 324.479 | ................................*X | 324.479 | | 294.466 | .............................**..*..**.**...*...*.......*X | 474.229 | | 466.029 | ..............................................X | 466.029 | | 461.600 | ..............................................*X | 461.600 | | 434.500 | ...........................................*X | 434.500 | | 419.900 | .........................................X | 419.900 | | 398.800 | .......................................*X | 398.800 | | 343.700 | ..................................X | 343.700 | | 323.200 | ................................X | 323.200 | | 295.200 | .............................***...X | 323.900 | | 322.669 | ................................*.*X | 333.737 | | 319.937 | ...............................**....**.....***..*X | 425.380 | | 404.237 | ........................................*X | 404.237 | | 403.037 | ........................................X | 403.037 | | 383.826 | ......................................***..*.....**......**....*.*X | 565.441 | | 543.761 | ......................................................X | 543.761 | | 455.783 | .............................................X | 455.783 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 22:39:32,680 | test | INFO | pool-4-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...*.**....**.*X | 15.801 | | 14.301 | .......*...*...**.....*..................*.......**.....*...X | 90.892 | | 87.214 | .................................................X | 87.214 | | 81.612 | ..............................................*....***X | 89.996 | | 74.712 | ..........................................***.......*........................**........X | 143.828 | | 139.428 | ...............................................................................X | 139.428 | | 120.416 | ....................................................................*X | 120.416 | | 101.205 | .........................................................*X | 102.205 | | 82.605 | ..............................................*...........*X | 101.200 | | 80.200 | .............................................X | 80.200 | | 68.700 | ......................................X | 68.700 | | 68.400 | ......................................**.............X | 91.583 | | 78.583 | ............................................*X | 78.583 | | 54.976 | ..............................X | 54.976 | | 39.001 | .....................X | 39.001 | | 34.101 | ..................*.............**....*.X | 64.500 | | 42.070 | .......................X | 42.070 | | 37.023 | ....................***....*...X | 48.800 | | 44.700 | ........................***...*.......**.......*...........X | 92.900 | | 87.600 | .................................................X | 87.600 | | 65.198 | ....................................*X | 65.198 | | 59.398 | .................................X | 59.398 | | 52.998 | .............................*...............**.*..............*....**.......X | 123.974 | | 111.343 | ...............................................................X | 111.343 | | 104.258 | ...........................................................*X | 104.258 | | 92.558 | ....................................................*......*.......**..............*X | 139.703 | | 126.167 | .......................................................................X | 126.167 | | 108.920 | .............................................................X | 108.920 | | 101.851 | .........................................................**..........**..............*...............*...*X | 173.534 | | 147.834 | ....................................................................................X | 147.834 | | 128.183 | ........................................................................*X | 128.183 | | 93.972 | .....................................................X | 93.972 | | 91.056 | ...................................................X | 91.056 | | 87.416 | .................................................**.........**.X | 105.650 | | 85.917 | ................................................*X | 85.917 | | 82.717 | ..............................................X | 82.717 | | 70.400 | .......................................*.......**..*X | 86.500 | | 82.700 | ..............................................X | 82.700 | | 75.900 | ..........................................*X | 75.900 | | 40.600 | ......................X | 40.600 | | 35.600 | ...................X | 35.600 | | 29.200 | ...............**............**....*...X | 61.900 | | 60.000 | .................................**...*....***X | 72.884 | | 67.184 | .....................................X | 67.184 | | 64.014 | ...................................*................***...........*X | 109.500 | | 69.700 | .......................................*X | 69.700 | | 56.800 | ...............................X | 56.800 | | 33.300 | ..................**...*X | 39.800 | | 39.600 | .....................***.X | 40.600 | | 40.500 | ......................X | 40.500 | | 36.200 | ...................*X | 36.200 | | 35.800 | ...................**.*.....**....*...X | 57.780 | | 57.507 | ................................*X | 57.507 | | 49.117 | ...........................*X | 49.117 | | 42.617 | .......................**....**.X | 50.994 | | 46.994 | ..........................*X | 46.994 | | 26.900 | ..............*X | 26.900 | | 13.800 | ......X | 13.800 | | 0.900 | *X | 0.900 | | 0.800 | *X | 0.800 | | 0.600 | ***X | 0.600 | | 0.400 | *********X | 1.601 | | 0.800 | X | 0.800 | | 0.700 | *****X | 1.099 | | 0.999 | *X | 0.999 | | 0.000 | *********..***X | 3.600 | | 2.000 | *X | 2.000 | | 1.900 | X | 1.900 | | 1.600 | *X | 1.600 | | 0.200 | *********.**X | 3.200 | | 2.500 | *X | 2.500 | | 0.500 | X | 0.500 | | 0.200 | ****.***X | 1.900 | | 0.100 | **X | 0.100 | | 0.000 | *****X | 1.301 | | 1.300 | **X | 1.300 | | 0.900 | ***.***X | 1.900 | | 0.400 | ******X | 1.200 | | 1.000 | *X | 1.000 | | 0.200 | *X | 0.200 | | 0.100 | **X | 0.100 | | 0.000 | ****.*X | 2.000 | | 1.700 | *X | 1.700 | | 1.100 | X | 1.100 | | 1.000 | ***X | 1.400 | | 0.400 | **X | 0.500 | | 0.400 | ***X | 0.900 | | 0.700 | *X | 0.700 | | 0.700 | X | 0.700 | | 0.500 | ********.**X | 2.700 | | 2.200 | *X | 2.200 | | 1.100 | ***X | 1.200 | | 1.000 | **X | 1.100 | | 1.100 | *X | 1.100 | | 0.600 | *X | 0.600 | | 0.500 | *X | 0.500 | | 0.300 | *****X | 1.000 | | 0.800 | *X | 0.800 | | 0.800 | *X | 0.800 | | 0.400 | ******X | 0.900 | | 0.800 | **.X | 2.400 | | 2.400 | **X | 2.900 | | 1.100 | *X | 1.100 | | 1.000 | X | 1.000 | | 0.600 | *****.******X | 3.098 | | 2.598 | *X | 2.598 | | 1.897 | *X | 1.897 | | 0.300 | ****.****X | 3.200 | | 1.800 | X | 1.800 | | 1.600 | *X | 1.600 | | 1.200 | **.***X | 2.200 | | 2.000 | X | 2.000 | | 1.700 | *X | 1.700 | | 1.100 | ******.***.*X | 4.102 | | 3.102 | *.*X | 3.502 | | 3.100 | *X | 3.299 | | 2.399 | *X | 2.399 | | 1.901 | *****.*X | 3.800 | | 3.200 | X | 3.200 | | 2.800 | **X | 3.000 | | 2.800 | **X | 3.000 | | 2.700 | **X | 2.700 | | 1.700 | *X | 1.700 | | 0.800 | X | 0.800 | | 0.200 | *******X | 1.600 | | 1.399 | **X | 1.399 | | 1.100 | ***.***.*X | 3.899 | | 2.899 | ***.X | 4.000 | | 3.401 | X | 3.401 | | 2.701 | *X | 2.701 | | 0.700 | *X | 0.700 | | 0.500 | X | 0.500 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 22:39:34,023 | test | INFO | pool-4-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *........**.......*...............**..*.X | 44.202 | | 44.099 | ................................**......*....X | 57.197 | | 45.000 | .................................**.*...*............*..*X | 68.512 | | 66.109 | .................................................X | 66.109 | | 64.894 | ................................................X | 64.894 | | 62.780 | ..............................................*..X | 65.499 | | 57.314 | ..........................................X | 57.314 | | 57.295 | ..........................................**..**....X | 65.194 | | 61.622 | .............................................**..............................**.................*X | 123.216 | | 120.716 | ..........................................................................................**..X | 122.600 | | 103.400 | .............................................................................X | 103.400 | | 101.600 | ............................................................................*X | 101.600 | | 77.800 | ..........................................................*.........X | 89.678 | | 81.878 | .............................................................**..................*...*....**..X | 118.370 | | 97.315 | ........................................................................X | 97.315 | | 88.815 | ..................................................................X | 88.815 | | 85.170 | ...............................................................**........................*............X | 131.677 | | 131.377 | ..................................................................................................*X | 131.377 | | 107.600 | ................................................................................X | 107.600 | | 107.300 | ................................................................................*X | 107.300 | | 86.600 | ................................................................**............**.............*.X | 120.600 | | 113.400 | .....................................................................................**.X | 115.492 | | 94.692 | ......................................................................X | 94.692 | | 84.492 | ...............................................................X | 84.492 | | 44.600 | ................................*X | 44.600 | | 0.000 | *******.***X | 1.500 | | 0.500 | **X | 0.500 | | 0.400 | X | 0.400 | | 0.000 | ****.***.....**...........*..........*.....**.*.***X | 45.604 | | 41.919 | ..............................X | 41.919 | | 41.207 | ..............................*X | 41.207 | | 39.807 | .............................X | 39.807 | | 39.700 | .............................***.*X | 41.400 | | 41.300 | ..............................X | 41.300 | | 39.000 | ............................**.*X | 40.200 | | 37.100 | ...........................*X | 37.100 | | 29.100 | .....................*X | 29.100 | | 21.200 | ...............*..*......**......*****.***....*....**X | 51.900 | | 39.000 | ............................*X | 39.000 | | 27.600 | ...................*.**....**..........**......*X | 55.000 | | 41.300 | ..............................X | 41.300 | | 39.900 | .............................*X | 39.900 | | 19.900 | ..............*X | 19.900 | | 14.000 | .........X | 14.000 | | 1.800 | *X | 1.800 | | 1.699 | X | 1.699 | | 0.699 | X | 0.699 | | 0.100 | *X | 0.100 | | 0.000 | *****X | 0.400 | | 0.200 | ****X | 0.200 | | 0.000 | *****.*.**X | 3.001 | | 1.999 | *****...X | 5.304 | | 5.204 | ..*..........**.........*..........*X | 42.600 | | 35.803 | ..........................**....*..*...........*X | 58.475 | | 57.922 | ..........................................*..**....*X | 65.716 | | 63.814 | ...............................................*X | 63.814 | | 57.414 | ..........................................**.*...**.**......*...*....**X | 80.392 | | 78.892 | ..........................................................*..*.*X | 82.108 | | 81.708 | .............................................................X | 81.708 | | 74.800 | .......................................................*X | 74.800 | | 68.187 | ..................................................*X | 68.187 | | 65.087 | ................................................*.**....*X | 72.316 | | 70.816 | ....................................................**X | 70.816 | | 65.100 | ................................................X | 65.100 | | 61.500 | .............................................*..*X | 63.796 | | 63.496 | ...............................................*X | 63.496 | | 53.384 | .......................................*......*...**..........*X | 78.730 | | 68.524 | ...................................................**X | 68.824 | | 65.800 | ................................................*...*X | 68.900 | | 63.700 | ...............................................*X | 63.700 | | 58.400 | ...........................................*...*.......**......X | 79.300 | | 76.900 | .........................................................*X | 76.900 | | 62.200 | ..............................................*X | 62.200 | | 49.100 | ....................................X | 49.100 | | 40.394 | .............................*..**.....**.......*..**.X | 63.201 | | 60.191 | ............................................**.......**............*......X | 92.477 | | 86.263 | ................................................................*X | 86.263 | | 64.863 | ................................................X | 64.863 | | 52.121 | ......................................*.....***.........*.........**X | 82.100 | | 81.311 | ............................................................X | 81.311 | | 78.411 | ..........................................................X | 78.411 | | 76.705 | .........................................................**........**...............*...........*X | 121.800 | | 115.400 | ......................................................................................X | 115.400 | | 99.900 | ..........................................................................*X | 99.900 | | 86.400 | ................................................................*X | 86.400 | | 60.000 | ............................................*X | 60.000 | | 47.380 | ..................................**.....*X | 53.680 | | 49.580 | ....................................*......*X | 57.130 | | 56.530 | .........................................*..*X | 59.230 | | 58.600 | ...........................................**.**X | 60.500 | | 52.100 | ......................................*X | 52.600 | | 51.600 | ......................................**.....*X | 59.100 | | 45.600 | .................................X | 45.600 | | 35.600 | ..........................X | 35.600 | | 15.800 | ..........*X | 15.800 | | 9.500 | ......*X | 9.500 | | 0.700 | X | 0.700 | | 0.600 | ****X | 0.600 | | 0.400 | **...**.**.X | 6.594 | | 6.495 | ...*X | 6.495 | | 6.294 | ...*X | 6.294 | | 4.103 | ..X | 4.103 | | 4.103 | ..*X | 4.103 | | 3.003 | .X | 3.003 | | 1.400 | X | 1.400 | | 1.200 | *X | 1.200 | | 1.100 | **X | 1.299 | | 1.299 | *X | 1.299 | | 1.299 | *X | 1.299 | | 0.600 | ****.....***.*X | 8.901 | | 5.200 | ..X | 5.200 | | 5.200 | ..X | 5.200 | | 3.500 | .*X | 3.500 | | 1.900 | X | 1.900 | | 1.400 | *X | 1.400 | | 0.900 | ******.*****.***...*X | 7.005 | | 6.805 | ....*X | 7.105 | | 4.001 | ..*X | 4.001 | | 1.800 | *X | 1.800 | | 0.600 | ***.*.****.***..X | 6.900 | | 6.500 | ...X | 6.500 | | 5.700 | ...*X | 5.700 | | 4.100 | ..*X | 4.100 | | 2.500 | X | 2.500 | | 2.200 | X | 2.200 | | 1.600 | *X | 1.600 | | 1.200 | **.X | 1.400 | | 1.300 | *X | 1.300 | | 0.700 | ****..*.**..***.***X | 8.599 | | 6.998 | ....***X | 7.500 | | 6.502 | ...X | 6.502 | | 6.302 | ...*X | 6.302 | | 2.700 | .*X | 2.700 | | 0.300 | X | 0.300 | +---------+---------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 22:39:34,171 | test | WARNING | Thread-5262 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:be5vh9-43-339000 update failed keys from task: [u'test_collections-809', u'test_collections-849', u'test_collections-601', u'test_collections-965', u'test_collections-644', u'test_collections-881', u'test_collections-561', u'test_collections-680', u'test_collections-931', u'test_collections-974', u'test_collections-773', u'test_collections-890'] 2021-05-02 22:39:34,174 | test | WARNING | Thread-5263 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:BS-V0eDy_eBRyPucfm%-43-339000 delete failed keys from task: [u'test_collections-466', u'test_collections-430'] 2021-05-02 22:39:34,177 | test | WARNING | Thread-5266 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:ARMTX8%7mdjjlzXR-43-336000 create failed keys from task: [u'test_collections-2672', u'test_collections-2730'] 2021-05-02 22:39:34,180 | test | WARNING | Thread-5272 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4xO2GcHI-43-341000 delete failed keys from task: [u'test_collections-63', u'test_collections-369', u'test_collections-234', u'test_collections-378', u'test_collections-154'] 2021-05-02 22:39:34,183 | test | WARNING | Thread-5273 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:f2MrbSa7Q-43-336000 delete failed keys from task: [u'test_collections-128'] 2021-05-02 22:39:34,194 | test | WARNING | Thread-5290 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1w0l-43-361000:bAvyhhv-43-367000 update failed keys from task: [u'test_collections-809', u'test_collections-849', u'test_collections-601', u'test_collections-644', u'test_collections-881', u'test_collections-561', u'test_collections-913', u'test_collections-933', u'test_collections-612', u'test_collections-931', u'test_collections-657', u'test_collections-698', u'test_collections-974', u'test_collections-773', u'test_collections-890', u'test_collections-550'] 2021-05-02 22:39:34,194 | test | WARNING | Thread-5291 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1w0l-43-361000:eIfxvbYI%3llNeO8qL6H-43-368000 delete failed keys from task: [u'test_collections-39'] 2021-05-02 22:39:34,203 | test | WARNING | Thread-5308 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1w0l-43-361000:Wbw0XG_SqFxrkgIw-%D-43-366000 update failed keys from task: [u'test_collections-926', u'test_collections-927', u'test_collections-916', u'test_collections-836', u'test_collections-865', u'test_collections-962', u'test_collections-820', u'test_collections-710', u'test_collections-873', u'test_collections-791'] 2021-05-02 22:39:34,204 | test | WARNING | Thread-5311 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1w0l-43-361000:5zEuoY3818-43-364000 delete failed keys from task: [u'test_collections-466', u'test_collections-430'] 2021-05-02 22:39:34,217 | test | WARNING | Thread-5338 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:XdvxYNFcKLcb-43-370000:T9PJd52Io6%zffMaJlH-43-386000 update failed keys from task: [u'test_collections-927', u'test_collections-905', u'test_collections-865', u'test_collections-962', u'test_collections-743', u'test_collections-820', u'test_collections-916', u'test_collections-836', u'test_collections-934', u'test_collections-710', u'test_collections-971', u'test_collections-873', u'test_collections-791'] 2021-05-02 22:39:34,219 | test | WARNING | Thread-5341 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:XdvxYNFcKLcb-43-370000:Pu5DCmjqYz-43-372000 update failed keys from task: [u'test_collections-829', u'test_collections-735', u'test_collections-659', u'test_collections-601', u'test_collections-696', u'test_collections-883', u'test_collections-593', u'test_collections-680'] 2021-05-02 22:39:34,316 | test | WARNING | Thread-5308 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1w0l-43-361000:Wbw0XG_SqFxrkgIw-%D-43-366000 delete failed keys from task: [u'test_collections-32', u'test_collections-448', u'test_collections-383', u'test_collections-281', u'test_collections-419', u'test_collections-459', u'test_collections-478', u'test_collections-138', u'test_collections-311', u'test_collections-213', u'test_collections-354', u'test_collections-491'] 2021-05-02 22:39:34,374 | test | WARNING | Thread-5272 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4xO2GcHI-43-341000 create failed keys from task: [u'test_collections-2867', u'test_collections-2672'] 2021-05-02 22:39:34,394 | test | WARNING | Thread-5396 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:S-43-358000 update failed keys from task: [u'test_collections-969', u'test_collections-967', u'test_collections-971'] 2021-05-02 22:39:34,398 | test | WARNING | Thread-5408 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:DeexDdTvq5d8ULw-43-274000 delete failed keys from task: [u'test_collections-1'] 2021-05-02 22:39:34,417 | test | WARNING | Thread-5437 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:1E8mx7ihnG5VKwm-14-392000:sb1myTZTpjAemJFo-14-513000 create failed keys from task: [u'test_collections-2291', u'test_collections-2379', u'test_collections-2535', u'test_collections-2969', u'test_collections-2244', u'test_collections-2452', u'test_collections-2301'] 2021-05-02 22:39:34,519 | test | WARNING | Thread-5338 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:XdvxYNFcKLcb-43-370000:T9PJd52Io6%zffMaJlH-43-386000 delete failed keys from task: [u'test_collections-307', u'test_collections-448', u'test_collections-342', u'test_collections-281', u'test_collections-419', u'test_collections-459', u'test_collections-478', u'test_collections-138', u'test_collections-311', u'test_collections-354', u'test_collections-297', u'test_collections-491'] 2021-05-02 22:39:34,579 | test | WARNING | Thread-5599 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:yDO7U4e7w327q5lx-15-682000:7dSVI6B-15-814000 create failed keys from task: [u'test_collections-4', u'test_collections-519', u'test_collections-359', u'test_collections-337', u'test_collections-118', u'test_collections-489', u'test_collections-97', u'test_collections-663'] 2021-05-02 22:39:34,581 | test | WARNING | Thread-5600 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:yDO7U4e7w327q5lx-15-682000:93M-15-815000 create failed keys from task: [u'test_collections-4', u'test_collections-519', u'test_collections-337', u'test_collections-359', u'test_collections-118', u'test_collections-489', u'test_collections-97', u'test_collections-663'] 2021-05-02 22:39:34,582 | test | WARNING | Thread-5601 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:4IWbHHjTd1-16-213000:KN4aMYAgjB%Trqx5a-16-342000 create failed keys from task: [u'test_collections-4', u'test_collections-519', u'test_collections-337', u'test_collections-359', u'test_collections-118', u'test_collections-489', u'test_collections-97', u'test_collections-663'] 2021-05-02 22:39:34,585 | test | WARNING | Thread-5603 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:STqVKrW-43-300000 update failed keys from task: [u'test_collections-4'] 2021-05-02 22:39:34,586 | test | WARNING | Thread-5604 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:wh-z7L-59u2XKK6u-43-303000 delete failed keys from task: [u'test_collections-1'] 2021-05-02 22:39:34,785 | test | WARNING | Thread-5557 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:PxR8vcl4-15-336000:VyNoVs0tAwD-15-484000 create failed keys from task: [u'test_collections-324', u'test_collections-314', u'test_collections-556', u'test_collections-265', u'test_collections-431', u'test_collections-220', u'test_collections-284'] 2021-05-02 22:39:34,785 | test | WARNING | Thread-5362 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:oIWtfhL_6M25FO-17-187000:P8RJoFhf-17-319000 create failed keys from task: [u'test_collections-3165'] 2021-05-02 22:39:34,789 | test | WARNING | Thread-5554 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:23ojEOJEMmWXsPI-15-12000:eVg_RQ6j3l1KfWdt-15-138000 create failed keys from task: [u'test_collections-324', u'test_collections-314', u'test_collections-556', u'test_collections-265', u'test_collections-431', u'test_collections-220', u'test_collections-284'] 2021-05-02 22:39:34,792 | test | WARNING | Thread-5556 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:PxR8vcl4-15-336000:0YRwoRy-15-483000 create failed keys from task: [u'test_collections-2380', u'test_collections-2210'] 2021-05-02 22:39:34,792 | test | WARNING | Thread-5376 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:jWgzINZCH-h32%MLjcX-43-355000 delete failed keys from task: [u'test_collections-39', u'test_collections-158', u'test_collections-147', u'test_collections-131'] 2021-05-02 22:39:34,792 | test | WARNING | Thread-5555 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:23ojEOJEMmWXsPI-15-12000:3RumrAMZUS-15-137000 create failed keys from task: [u'test_collections-1509', u'test_collections-1499', u'test_collections-2156'] 2021-05-02 22:39:34,789 | test | WARNING | Thread-5375 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:944S4YEtlOi-43-350000 update failed keys from task: [u'test_collections-503', u'test_collections-566', u'test_collections-564', u'test_collections-521'] 2021-05-02 22:39:34,789 | test | WARNING | Thread-5380 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:urjiC4%u8%J-43-349000 update failed keys from task: [u'test_collections-626', u'test_collections-546', u'test_collections-555', u'test_collections-501', u'test_collections-589', u'test_collections-663', u'test_collections-543', u'test_collections-675'] 2021-05-02 22:39:49,065 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 22:41:00,950 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-02 22:41:00,951 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-02 22:41:19,194 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-02 22:41:19,196 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-02 22:41:44,857 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-02 22:41:44,858 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-02 22:42:32,786 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-02 22:42:51,388 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1610000134 seconds to complete 2021-05-02 22:42:51,411 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2374.486s During the test, Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_2 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_3', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 3, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-02 22:43:01,414 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-02 22:43:02,392 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-02 22:43:02,927 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb969f type=client open_count=1 channel=[id: 0x41d400e6, 0.0.0.0/0.0.0.0:53257] timeout=300.0>) 2021-05-02 22:43:08,378 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a0 type=client open_count=1 channel=[id: 0xb314dcc9, 0.0.0.0/0.0.0.0:37244] timeout=300.0>) 2021-05-02 22:43:10,400 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a1 type=client open_count=1 channel=[id: 0xd086e5c4, 0.0.0.0/0.0.0.0:47195] timeout=300.0>) 2021-05-02 22:43:12,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a2 type=client open_count=1 channel=[id: 0xee232a97, 0.0.0.0/0.0.0.0:54243] timeout=300.0>) 2021-05-02 22:43:14,444 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a3 type=client open_count=1 channel=[id: 0x05a898a7, 0.0.0.0/0.0.0.0:57452] timeout=300.0>) 2021-05-02 22:43:16,467 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a4 type=client open_count=1 channel=[id: 0xa7de7c85, 0.0.0.0/0.0.0.0:38793] timeout=300.0>) 2021-05-02 22:43:18,489 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a5 type=client open_count=1 channel=[id: 0x935c51bf, 0.0.0.0/0.0.0.0:52535] timeout=300.0>) 2021-05-02 22:43:20,512 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a6 type=client open_count=1 channel=[id: 0x167844ee, 0.0.0.0/0.0.0.0:57719] timeout=300.0>) 2021-05-02 22:43:22,535 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb96a7 type=client open_count=1 channel=[id: 0x306b0923, 0.0.0.0/0.0.0.0:47427] timeout=300.0>) 2021-05-02 22:43:24,559 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96a8 type=client open_count=1 channel=[id: 0x63597f2d, 0.0.0.0/0.0.0.0:37850] timeout=300.0>) 2021-05-02 22:43:31,931 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96a9 type=client open_count=1 channel=[id: 0x32a25c2e, 0.0.0.0/0.0.0.0:52163] timeout=300.0>) 2021-05-02 22:43:33,956 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96aa type=client open_count=1 channel=[id: 0xe68eaacf, 0.0.0.0/0.0.0.0:36247] timeout=300.0>) 2021-05-02 22:43:35,979 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96ab type=client open_count=1 channel=[id: 0xd24b383c, 0.0.0.0/0.0.0.0:45313] timeout=300.0>) 2021-05-02 22:43:38,006 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96ac type=client open_count=1 channel=[id: 0x46515759, 0.0.0.0/0.0.0.0:46175] timeout=300.0>) 2021-05-02 22:43:40,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96ad type=client open_count=1 channel=[id: 0x0bcbdd2a, 0.0.0.0/0.0.0.0:38719] timeout=300.0>) 2021-05-02 22:43:42,048 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96ae type=client open_count=1 channel=[id: 0x2a9abdfb, 0.0.0.0/0.0.0.0:55405] timeout=300.0>) 2021-05-02 22:43:44,071 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96af type=client open_count=1 channel=[id: 0x790e23b8, 0.0.0.0/0.0.0.0:38385] timeout=300.0>) 2021-05-02 22:43:46,092 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb96b0 type=client open_count=1 channel=[id: 0x55fdd009, 0.0.0.0/0.0.0.0:38520] timeout=300.0>) 2021-05-02 22:43:48,114 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b1 type=client open_count=1 channel=[id: 0x524af365, 0.0.0.0/0.0.0.0:34971] timeout=300.0>) 2021-05-02 22:43:55,589 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b2 type=client open_count=1 channel=[id: 0x507af50f, 0.0.0.0/0.0.0.0:33943] timeout=300.0>) 2021-05-02 22:43:57,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b3 type=client open_count=1 channel=[id: 0x3e6a4a73, 0.0.0.0/0.0.0.0:35385] timeout=300.0>) 2021-05-02 22:43:59,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b4 type=client open_count=1 channel=[id: 0x21ef418a, 0.0.0.0/0.0.0.0:33372] timeout=300.0>) 2021-05-02 22:44:01,651 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b5 type=client open_count=1 channel=[id: 0x2c91fc77, 0.0.0.0/0.0.0.0:47031] timeout=300.0>) 2021-05-02 22:44:03,674 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b6 type=client open_count=1 channel=[id: 0xd391d279, 0.0.0.0/0.0.0.0:42762] timeout=300.0>) 2021-05-02 22:44:05,697 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b7 type=client open_count=1 channel=[id: 0x01df6405, 0.0.0.0/0.0.0.0:35717] timeout=300.0>) 2021-05-02 22:44:07,723 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b8 type=client open_count=1 channel=[id: 0x24d41806, 0.0.0.0/0.0.0.0:40584] timeout=300.0>) 2021-05-02 22:44:09,746 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96b9 type=client open_count=1 channel=[id: 0x5414ae50, 0.0.0.0/0.0.0.0:33071] timeout=300.0>) 2021-05-02 22:44:11,769 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96ba type=client open_count=1 channel=[id: 0xf95413c6, 0.0.0.0/0.0.0.0:56555] timeout=300.0>) 2021-05-02 22:44:13,792 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb96bb type=client open_count=1 channel=[id: 0x2101becd, 0.0.0.0/0.0.0.0:55261] timeout=300.0>) 2021-05-02 22:44:15,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96bc type=client open_count=1 channel=[id: 0x1d68a14e, 0.0.0.0/0.0.0.0:35446] timeout=300.0>) 2021-05-02 22:44:23,970 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96bd type=client open_count=1 channel=[id: 0x282c6756, 0.0.0.0/0.0.0.0:60375] timeout=300.0>) 2021-05-02 22:44:25,993 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96be type=client open_count=1 channel=[id: 0xac6fd7b7, 0.0.0.0/0.0.0.0:49493] timeout=300.0>) 2021-05-02 22:44:28,017 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96bf type=client open_count=1 channel=[id: 0xef899f85, 0.0.0.0/0.0.0.0:47464] timeout=300.0>) 2021-05-02 22:44:30,062 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96c0 type=client open_count=1 channel=[id: 0x6e580866, 0.0.0.0/0.0.0.0:38985] timeout=300.0>) 2021-05-02 22:44:32,085 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96c1 type=client open_count=1 channel=[id: 0x2ab39546, 0.0.0.0/0.0.0.0:34981] timeout=300.0>) 2021-05-02 22:44:34,117 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96c2 type=client open_count=1 channel=[id: 0xb98cbed1, 0.0.0.0/0.0.0.0:48255] timeout=300.0>) 2021-05-02 22:44:36,141 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96c3 type=client open_count=1 channel=[id: 0x5cd69975, 0.0.0.0/0.0.0.0:35637] timeout=300.0>) 2021-05-02 22:44:38,167 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb96c4 type=client open_count=1 channel=[id: 0x22fd452e, 0.0.0.0/0.0.0.0:42346] timeout=300.0>) 2021-05-02 22:44:40,194 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb96c5 type=client open_count=1 channel=[id: 0xd47b614e, 0.0.0.0/0.0.0.0:59033] timeout=300.0>) 2021-05-02 22:44:45,276 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb96c6 type=client open_count=1 channel=[id: 0x26167fb2, 0.0.0.0/0.0.0.0:42207] timeout=300.0>) 2021-05-02 22:44:47,295 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb96c7 type=client open_count=1 channel=[id: 0x92cce902, 0.0.0.0/0.0.0.0:40909] timeout=300.0>) 2021-05-02 22:44:49,336 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb96c8 type=client open_count=1 channel=[id: 0x1acaec24, 0.0.0.0/0.0.0.0:37531] timeout=300.0>) 2021-05-02 22:44:51,358 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb96c9 type=client open_count=1 channel=[id: 0x525144ff, 0.0.0.0/0.0.0.0:57315] timeout=300.0>) 2021-05-02 22:44:53,380 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb96ca type=client open_count=1 channel=[id: 0x28691769, 0.0.0.0/0.0.0.0:48547] timeout=300.0>) 2021-05-02 22:44:55,398 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 22:45:12,500 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-02 22:45:27,194 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-02 22:45:27,221 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-02 22:45:27,270 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-02 22:46:00,227 | test | INFO | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 14.3398131785 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-02 22:46:00,257 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 22:46:10,352 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-02 22:46:10,387 | test | INFO | pool-5-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1600000858 sec 2021-05-02 22:46:10,736 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-02 22:46:52,555 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-02 22:46:52,921 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 632454080 | 68877252 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 549986624 | 97430903 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 697945216 | 49281807 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-02 22:46:52,921 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-02 22:47:46,361 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 22:47:46,377 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 23:03:38,812 | test | INFO | pool-5-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.............*............................................................**.......................*....X | 200.100 | | 79.300 | ......................................*X | 79.300 | | 34.000 | ...............*X | 34.000 | | 0.000 | ************************************.....*................**......*X | 55.494 | | 48.294 | .......................X | 48.294 | | 16.500 | .......X | 16.500 | | 4.500 | .*X | 4.500 | | 0.100 | X | 0.100 | | 0.000 | ******************************************************************************************.*....*.*X | 13.199 | | 9.599 | ...X | 9.599 | | 2.400 | X | 2.400 | | 0.000 | ***********************************************************************************************************************************************************************************************************************************************************************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 23:03:39,174 | test | INFO | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **........**....................**.............................*...........................................X | 1298.138 | | 1248.438 | ...............................................................................................*X | 1248.438 | | 1091.438 | ...................................................................................X | 1091.438 | | 855.544 | ................................................................**.*X | 858.201 | | 808.201 | .............................................................X | 808.201 | | 710.957 | .....................................................X | 710.957 | | 652.031 | .................................................**..*...**...*X | 756.553 | | 737.018 | .......................................................X | 737.018 | | 735.818 | .......................................................*X | 735.818 | | 731.524 | .......................................................*.X | 749.052 | | 742.752 | ........................................................*X | 742.752 | | 701.156 | .....................................................*X | 701.156 | | 633.556 | ...............................................X | 633.556 | | 490.656 | ....................................*X | 490.656 | | 455.234 | ..................................***..**X | 490.891 | | 475.991 | ...................................X | 475.991 | | 454.791 | ..................................*X | 454.791 | | 442.097 | .................................*X | 442.097 | | 424.057 | ...............................***.*...*........**...*X | 620.664 | | 611.664 | ..............................................X | 611.664 | | 606.900 | .............................................**.*...*.*X | 670.401 | | 670.066 | ..................................................*X | 670.066 | | 654.866 | .................................................X | 654.866 | | 650.266 | .................................................*X | 650.266 | | 635.770 | ...............................................X | 635.770 | | 605.370 | .............................................X | 605.370 | | 548.717 | .........................................*X | 548.717 | | 531.300 | .......................................*X | 531.300 | | 496.503 | .....................................*X | 496.503 | | 463.850 | ..................................*X | 463.850 | | 439.088 | ................................X | 439.088 | | 398.438 | .............................*X | 398.438 | | 395.938 | .............................*X | 395.938 | | 371.947 | ...........................*.***...**..*.**.*.......*..**.X | 602.244 | | 573.544 | ...........................................**..**......**X | 682.400 | | 664.900 | ..................................................X | 664.900 | | 650.900 | .................................................X | 650.900 | | 613.879 | ..............................................**.*X | 626.379 | | 583.250 | ...........................................**X | 583.250 | | 554.250 | .........................................*.*......**.*X | 653.283 | | 621.445 | ..............................................X | 621.445 | | 526.022 | .......................................X | 526.022 | | 468.203 | ...................................*X | 468.203 | | 441.872 | .................................*X | 441.872 | | 400.642 | .............................*..**...*......**X | 540.100 | | 539.800 | ........................................X | 539.800 | | 487.400 | ....................................****.*...**..X | 564.091 | | 563.591 | ..........................................*..**.*X | 602.224 | | 570.048 | ..........................................**X | 570.048 | | 517.548 | ......................................X | 517.548 | | 433.848 | ................................X | 433.848 | | 310.300 | ......................*X | 310.300 | | 275.100 | ....................**.*..*...**.**....***.*..**..**.....**X | 556.738 | | 554.038 | .........................................X | 554.038 | | 527.658 | .......................................X | 527.658 | | 506.186 | .....................................**.*X | 509.686 | | 496.486 | .....................................*X | 501.552 | | 484.652 | ....................................**..*X | 514.052 | | 501.000 | .....................................*.*X | 513.700 | | 496.100 | .....................................X | 496.100 | | 495.900 | .....................................*X | 495.900 | | 470.000 | ...................................*X | 470.000 | | 424.035 | ...............................**.**..***..**..*.........*.*X | 646.170 | | 631.070 | ...............................................*X | 631.070 | | 583.436 | ...........................................X | 583.436 | | 553.748 | .........................................**.*.**.X | 588.416 | | 588.016 | ............................................X | 588.016 | | 549.016 | .........................................**X | 549.016 | | 524.128 | .......................................X | 524.128 | | 466.628 | ..................................X | 466.628 | | 396.828 | .............................*X | 396.828 | | 369.255 | ...........................**...***...*.*.....***X | 527.909 | | 439.968 | ................................**X | 439.968 | | 422.468 | ...............................**.....**.*...**.X | 554.500 | | 542.300 | ........................................*X | 542.300 | | 507.700 | ......................................X | 507.700 | | 428.440 | ................................*X | 428.440 | | 414.140 | ..............................X | 414.140 | | 405.540 | ..............................*.***..*....**..*X | 521.886 | | 506.777 | ......................................*X | 506.777 | | 506.392 | ......................................**.*....**.*.X | 605.462 | | 582.203 | ...........................................**X | 582.203 | | 568.272 | ..........................................X | 568.272 | | 557.472 | .........................................X | 557.472 | | 555.328 | .........................................*X | 555.328 | | 534.800 | ........................................*X | 534.800 | | 521.600 | .......................................X | 521.600 | | 472.945 | ...................................X | 472.945 | | 456.145 | ..................................*X | 456.145 | | 435.145 | ................................*X | 435.145 | | 422.851 | ...............................*.**.X | 444.751 | | 440.100 | ................................*X | 440.100 | | 426.900 | ...............................*.*.....**....**....*....*X | 650.803 | | 644.903 | ................................................*X | 644.903 | | 626.252 | ...............................................X | 626.252 | | 606.315 | .............................................*X | 606.315 | | 596.199 | ............................................X | 596.199 | | 587.399 | ............................................X | 587.399 | | 571.012 | ..........................................*X | 571.012 | | 555.332 | .........................................*X | 555.332 | | 524.432 | .......................................X | 524.432 | | 449.104 | .................................*X | 449.104 | | 422.000 | ...............................*X | 422.000 | | 390.300 | .............................*X | 390.300 | | 385.161 | ............................**.*..**.**..*..*X | 491.684 | | 486.836 | ....................................X | 486.836 | | 484.625 | ....................................*..**..**..*..*.*X | 609.206 | | 599.460 | .............................................X | 599.460 | | 574.554 | ...........................................*X | 574.554 | | 552.339 | .........................................X | 552.339 | | 523.439 | .......................................X | 523.439 | | 467.239 | ..................................*X | 467.239 | | 461.900 | ..................................*X | 461.900 | | 429.670 | ................................**X | 429.670 | | 382.370 | ............................X | 382.370 | | 372.027 | ...........................X | 372.027 | | 347.527 | .........................**..**.....**.***...**..**..**.*X | 547.095 | | 534.386 | ........................................X | 534.386 | | 519.886 | .......................................*X | 519.886 | | 510.086 | ......................................**.....**.*X | 589.000 | | 554.800 | .........................................X | 554.800 | | 456.700 | ..................................*X | 456.700 | | 421.679 | ...............................*X | 421.679 | | 410.597 | ..............................*.*.**..*.***X | 478.483 | | 463.880 | ..................................*.*.**..*..*X | 539.872 | | 539.653 | ........................................**.X | 545.716 | | 536.018 | ........................................X | 536.018 | | 531.101 | .......................................*X | 531.101 | | 504.664 | .....................................X | 504.664 | | 450.464 | .................................X | 450.464 | | 405.433 | ..............................*X | 405.433 | | 375.033 | ...........................X | 375.033 | | 332.309 | ........................**..**...*......**.*X | 480.638 | | 462.338 | ..................................X | 462.338 | | 431.279 | ................................*.**..*..X | 494.990 | | 489.090 | ....................................**..*............**.....X | 739.698 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 23:03:41,137 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **************************************************************************************************************************************************...*...............*...............**.......................*......................................**......X | 83.200 | | 80.000 | ...............................................................................................*X | 80.000 | | 63.606 | ...........................................................................X | 63.606 | | 49.506 | ..........................................................*X | 49.506 | | 31.106 | ....................................X | 31.106 | | 2.700 | ..*X | 2.700 | | 0.000 | ***********************************************************************************************************************************************************************************************************************************************...*..................*...............................................**...................**...**X | 75.602 | | 63.595 | ...........................................................................X | 63.595 | | 41.995 | .................................................X | 41.995 | | 6.600 | ......*X | 6.600 | | 0.000 | **************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-02 23:03:41,515 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 47.7769008156 | 3693387776 | 2526375936 | 0 / 3909087232 | 125425 / 251577 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 47.4402730375 | 3693387776 | 2594549760 | 0 / 3909087232 | 125649 / 253790 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 47.5575084001 | 6078930944 | 4715335680 | 0 / 3758092288 | 125775 / 246695 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 49.383033419 | 4201836544 | 3060760576 | 0 / 3758092288 | 125272 / 252087 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-02 23:03:55,207 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 23:05:37,187 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155473936 | 231900264 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155366928 | 275710411 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 434777312 | 468190933 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-02 23:05:37,190 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-02 23:05:48,506 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during swap_rebalance 2021-05-02 23:05:48,506 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : swap_rebalance 2021-05-02 23:05:56,119 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 23:05:56,398 | test | INFO | pool-5-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 22.010178117 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 14.1768292683 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 14.2496210207 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 0.728094401205 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 12.2272957889 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-02 23:05:56,480 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:06:01,660 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:06:05,980 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 23:06:07,898 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.22 2021-05-02 23:06:11,098 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:scope1:A5D%iy0zrcSp3tEwEAU-10-830000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:11,101 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ru8z8PvywzIbxa-10-840000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:11,102 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:scope1:nGItBB8n1DXPDC-10-829000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:11,107 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ihvs5Co6fc_N%-10-842000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:11,107 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:Eq%PnX6J-10-839000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:11,107 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:wIq5FBdocmK-10-839000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:11,108 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:s%ZD-10-841000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:11,115 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:yO4bAhC3Tkv7--10-841000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:13,105 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 2.95 2021-05-02 23:06:18,292 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 5.4 2021-05-02 23:06:23,622 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 7.86 2021-05-02 23:06:25,790 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:wIq5FBdocmK-10-839000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,792 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:yO4bAhC3Tkv7--10-841000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,792 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ru8z8PvywzIbxa-10-840000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,793 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ihvs5Co6fc_N%-10-842000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,795 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:Eq%PnX6J-10-839000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,798 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:scope1:nGItBB8n1DXPDC-10-829000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,798 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket1:scope1:A5D%iy0zrcSp3tEwEAU-10-830000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,799 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:RiFZyaWh1xlhtBTsC-10-837000:s%ZD-10-841000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:25,976 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:06:25,977 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:06:28,750 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 10.51 2021-05-02 23:06:33,881 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 13.0 2021-05-02 23:06:39,026 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 16.39 2021-05-02 23:06:42,538 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-126 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:42,539 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-126 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:42,539 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-126 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:42,539 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-126 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-162 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:42,539 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-117 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:42,542 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-117 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:42,566 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-48 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-117 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:42,569 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-48 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-117 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:44,144 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 19.88 2021-05-02 23:06:49,290 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 23.07 2021-05-02 23:06:54,020 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-107 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:54,022 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-107 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:54,022 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-62 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:54,023 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-62 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:54,400 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 26.34 2021-05-02 23:06:56,769 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:56,769 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:56,769 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:56,769 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:56,770 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-300 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-290 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:56,770 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-300 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-290 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:56,772 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-300 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-290 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:56,772 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-300 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-290 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:06:59,536 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 29.59 2021-05-02 23:07:00,586 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:00,588 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:00,595 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:00,595 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:00,596 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:00,596 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:00,601 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:00,602 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,522 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,522 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,523 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,523 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,526 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,528 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,530 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,532 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,536 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,538 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,539 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:01,540 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,220 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,220 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,220 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,220 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,233 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,233 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,240 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,242 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,242 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,243 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,252 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:02,253 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,832 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1176 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1036 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,834 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1176 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1036 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,838 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1176 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1036 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,838 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1176 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1036 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,838 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1176 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1036 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,838 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1176 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1036 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,838 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1176 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1036 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,839 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1176 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1036 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,842 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1176 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1036 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,841 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1176 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1036 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,841 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1176 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1036 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:03,842 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1176 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1036 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:07:04,635 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 32.72 2021-05-02 23:07:09,799 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 23:07:11,974 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-02 23:07:11,977 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:07:11,979 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:07:11,979 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-02 23:07:11,980 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:07:11,980 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:07:11,980 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:07:11,982 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:07:11,982 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:07:11,983 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:07:11,983 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:07:11,983 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:07:11,984 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:07:11,984 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:07:11,986 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-02 23:07:11,986 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-02 23:07:11,990 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-02 23:07:11,992 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 10 docs from 0 to 5000 2021-05-02 23:07:11,993 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-02 23:07:11,993 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-02 23:07:14,890 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 23:07:20,655 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 34.62 2021-05-02 23:07:25,835 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 38.11 2021-05-02 23:07:31,032 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 40.79 2021-05-02 23:07:36,219 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 43.42 2021-05-02 23:07:41,385 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 46.2 2021-05-02 23:07:46,546 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 49.03 2021-05-02 23:07:51,704 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 51.85 2021-05-02 23:07:56,842 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 54.47 2021-05-02 23:08:01,980 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 57.22 2021-05-02 23:08:07,118 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.23 2021-05-02 23:08:12,260 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 63.25 2021-05-02 23:08:17,371 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.58 2021-05-02 23:08:22,509 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 23:08:27,647 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 23:08:32,391 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:32,529 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-635 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:33,263 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:_default:F-10-853000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:33,733 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.73 2021-05-02 23:08:33,799 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:34,131 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:5C-10-865000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:34,598 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-546 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-635 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:34,615 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:5C-10-865000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-546 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-635 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:34,673 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:2IMdZX4328EDgEpQWx-10-854000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-234 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:35,168 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:Jx4QNINElmqCAMNdF-10-853000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-956 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:37,844 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-145 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:38,118 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-234 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-271 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:38,128 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2855 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:38,128 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2855 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:38,207 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-637 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-546 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-503 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:38,966 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 71.58 2021-05-02 23:08:40,145 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:40,227 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:_default:F-10-853000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-234 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-243 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-271 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:40,394 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:2IMdZX4328EDgEpQWx-10-854000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-913 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-925 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,211 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-271 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,213 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-79 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,216 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-79 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,265 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:5C-10-865000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-709 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,296 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:_default:F-10-853000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-869 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-782 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,299 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:F-10-853000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2734 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,301 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:F-10-853000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2734 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:41,387 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:5C-10-865000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-271 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:43,871 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1657 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:43,871 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1657 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:43,947 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:2IMdZX4328EDgEpQWx-10-854000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-493 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:44,104 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:9r8y-4-677000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:44,105 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:9r8y-4-677000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-172 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:44,161 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 74.46 2021-05-02 23:08:45,010 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-349 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-253 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-231 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:45,446 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:5C-10-865000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:45,457 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-748 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-878 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-793 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:45,473 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-709 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:45,500 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-439 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-456 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:45,506 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:_default:F-10-853000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-456 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:46,463 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-166 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:46,464 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-166 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-121 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:46,476 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2816 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2726 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:46,476 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2726 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:47,062 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:5C-10-865000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:47,131 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-253 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:47,134 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-244 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-253 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:48,842 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-537 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-533 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-552 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:48,845 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-537 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-533 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-552 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:49,003 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:49,335 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 77.38 2021-05-02 23:08:49,819 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:50,898 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:CqtEagom7q-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-537 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-533 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-552 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:51,460 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:_default:F-10-853000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:51,477 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:51,788 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:51,805 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-413 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-485 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:51,891 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:CqtEagom7q-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-131 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-165 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:51,895 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-932 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:52,710 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-932 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,414 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1946 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,415 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1946 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,431 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-248 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-267 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-240 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,440 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-267 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-222 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-240 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,503 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:9r8y-4-677000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,505 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:9r8y-4-677000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-223 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,506 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:53,506 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-223 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:54,533 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.83 2021-05-02 23:08:54,954 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-716 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-895 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:55,114 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-716 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-895 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:55,858 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2874 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:55,861 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2874 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:55,865 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2765 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:55,867 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2765 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:55,960 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:CqtEagom7q-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-838 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-859 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-895 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-892 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:56,591 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope1:kRpj-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-108 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-128 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:56,591 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:scope1:3VW5-10-857000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-108 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-128 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:56,841 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:3VW5-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-557 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-532 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:56,861 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:scope1:kRpj-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-557 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-532 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:58,440 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:CqtEagom7q-10-858000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-221 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-341 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:58,503 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:NgAs-5-147000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-225 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:58,503 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:NgAs-5-147000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-225 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:59,265 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2050 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2068 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:59,266 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2050 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2068 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:59,476 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-428 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-422 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-496 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:59,480 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-428 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-422 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-496 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:08:59,759 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 82.36 2021-05-02 23:08:59,845 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:KL3O%n97isrbjno-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-916 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:00,891 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:CqtEagom7q-10-858000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-916 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:00,894 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:6M-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-916 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:01,049 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope1:kRpj-10-858000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-274 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-350 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:01,052 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:scope1:3VW5-10-857000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-274 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-350 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:02,091 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:3VW5-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-713 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-833 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-756 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-886 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:02,507 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:CqtEagom7q-10-858000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-425 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-434 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-422 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-499 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:02,545 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:scope1:kRpj-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-713 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-833 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-756 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-886 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:02,759 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-509 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:02,762 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-509 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:03,019 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-138 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:03,020 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-138 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:03,230 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:03,230 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2517 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2573 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2542 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:03,230 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:03,232 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2573 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:04,907 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 84.85 2021-05-02 23:09:05,082 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:05,084 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:05,088 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:05,089 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:05,155 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:05,229 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:05,230 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2600 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2622 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:05,276 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-690 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:06,051 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:q-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-674 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:06,226 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:scope1:q-10-855000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2684 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:06,227 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:scope1:q-10-855000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2535 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2684 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:06,239 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:3VW5-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-979 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:06,282 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:q-10-855000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-106 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:06,341 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope1:kRpj-10-858000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-434 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-466 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:06,341 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:scope1:3VW5-10-857000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-434 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-466 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:07,026 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:07,036 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:07,375 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:dlKrjx-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-627 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:07,727 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:07,730 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:08,236 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-258 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:09,213 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2788 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:09,213 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2788 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:10,017 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 87.46 2021-05-02 23:09:10,210 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-778 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-891 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:10,275 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:q-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-765 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:10,568 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:scope1:-B0gYMS-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-54 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:10,578 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:dlKrjx-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-54 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:10,865 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:q-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-219 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:11,479 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:11,479 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:11,714 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:dlKrjx-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-765 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:12,523 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-418 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-430 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:12,523 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:lL1HnFn-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-418 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-430 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-450 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:12,523 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-418 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-430 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:12,686 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:NgAs-5-147000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-639 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:12,687 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:NgAs-5-147000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-639 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:14,325 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:scope1:-B0gYMS-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-765 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:14,822 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-502 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:14,996 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:q-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-459 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:15,000 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:dlKrjx-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-343 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:15,144 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 89.76 2021-05-02 23:09:15,934 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:dlKrjx-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-969 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:16,573 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:vx1gX_7qwnoKAOF-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-61 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-178 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-142 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:16,612 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:scope1:vx1gX_7qwnoKAOF-10-855000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:16,614 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:scope1:vx1gX_7qwnoKAOF-10-855000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2570 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:16,970 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1023 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:16,971 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1023 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:17,072 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:scope1:KyAXYneDVJMbur-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-17 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-61 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-178 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-142 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:17,456 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:vx1gX_7qwnoKAOF-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:17,456 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:scope1:KyAXYneDVJMbur-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:17,661 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:scope1:-B0gYMS-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-343 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:19,091 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:KyAXYneDVJMbur-10-855000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:19,092 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:KyAXYneDVJMbur-10-855000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:19,117 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:scope1:collection_1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:19,118 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:scope1:collection_1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2826 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:19,509 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-337 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:19,790 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:dlKrjx-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-453 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:20,221 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:Z-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-663 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:20,278 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:20,387 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 92.17 2021-05-02 23:09:20,839 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:Z-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-111 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:21,516 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:UeTdD-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-75 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-179 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-82 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:21,937 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:UeTdD-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-590 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:23,013 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:scope1:KyAXYneDVJMbur-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-372 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:23,016 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:vx1gX_7qwnoKAOF-10-855000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-372 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:23,280 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope1:Z-10-856000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:23,282 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope1:Z-10-856000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2551 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2664 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:23,724 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-462 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:23,855 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:23,857 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2492 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:24,072 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1349 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:24,072 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-945 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:24,073 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1349 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:24,375 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:KyAXYneDVJMbur-10-855000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:24,378 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:scope1:KyAXYneDVJMbur-10-855000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2746 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:25,025 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:scope1:H4p-5kk-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-596 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:25,068 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:scope1:H4p-5kk-10-859000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:25,069 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:scope1:H4p-5kk-10-859000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2569 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:25,619 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 94.68 2021-05-02 23:09:25,875 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:UeTdD-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-355 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-268 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-364 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-374 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:25,920 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:Z-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-853 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:27,194 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:S1NQn87aapfAiv_vtnK-5-538000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3072 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:27,196 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:S1NQn87aapfAiv_vtnK-5-538000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3072 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:27,302 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:27,302 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2613 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2615 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2783 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2785 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:27,463 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:2H3a3w53MQrlPaQTR4S-10-859000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-47 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:27,786 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1434 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1432 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:27,789 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:75hRg7m-4-676000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1434 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1432 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:28,071 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:scope1:UeTdD-10-856000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:28,072 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:scope1:UeTdD-10-856000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:28,971 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:scope1:H4p-5kk-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-776 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-754 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:29,424 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:Z-10-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-355 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-364 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:29,849 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-681 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:30,051 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:30,052 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2501 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:30,727 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 97.15 2021-05-02 23:09:31,523 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:scope1:9FEgAfFzv_8XkE30U-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:31,664 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:scope1:9FEgAfFzv_8XkE30U-10-856000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:32,236 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1472 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:32,237 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1472 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:32,601 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:scope1:9FEgAfFzv_8XkE30U-10-856000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:32,601 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:scope1:9FEgAfFzv_8XkE30U-10-856000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:33,434 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:scope1:H4p-5kk-10-859000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-928 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:33,454 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:scope1:H4p-5kk-10-859000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:34,954 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:scope1:2H3a3w53MQrlPaQTR4S-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-262 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:34,954 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-356 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-332 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:35,729 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:scope1:2H3a3w53MQrlPaQTR4S-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-871 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:35,841 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 99.84 2021-05-02 23:09:35,976 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:Z-10-856000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-419 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,368 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:scope1:ewQTD-10-855000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,380 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,381 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2891 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,424 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:9r8y-4-677000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-714 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-684 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,424 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:YyH70SSha2-4-477000:9r8y-4-677000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-714 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-684 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,509 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:NORvTSE7XgYIZeJOov-10-858000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-70 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,563 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:scope1:RWY022TdyYwK4GIe2ShY-5-752000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,565 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:scope1:RWY022TdyYwK4GIe2ShY-5-752000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,663 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-788 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,865 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1742 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:36,865 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1742 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:37,559 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:ewQTD-10-855000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:37,561 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope1:ewQTD-10-855000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:09:40,950 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-02 23:09:46,068 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-02 23:09:56,637 | test | INFO | pool-5-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x115f74 type=client open_count=1 channel=[id: 0x2a1ff99f, 0.0.0.0/0.0.0.0:44153] timeout=300.0>) 2021-05-02 23:09:56,720 | test | ERROR | pool-5-thread-19 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x115f93 type=client open_count=1 channel=[id: 0xd5478a6b, 0.0.0.0/0.0.0.0:47391] timeout=300.0>) 2021-05-02 23:09:58,734 | test | ERROR | pool-5-thread-19 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 23:10:01,085 | test | INFO | pool-5-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 244.687000036 sec 2021-05-02 23:10:22,221 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-02 23:10:22,223 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 12 docs from 0 to 5000 2021-05-02 23:10:22,237 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:10:22,239 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-02 23:10:22,240 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-02 23:10:22,242 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-02 23:10:22,243 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-02 23:10:22,243 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:10:22,243 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-02 23:10:22,244 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-02 23:10:22,246 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:22,256 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-02 23:10:22,256 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-02 23:10:22,257 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-02 23:10:22,259 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-02 23:10:22,259 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 13 docs from 500 to 980 2021-05-02 23:10:22,259 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 11 docs from 0 to 500 2021-05-02 23:10:22,260 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:10:40,375 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-02 23:10:40,375 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 10 docs from 0 to 5000 2021-05-02 23:10:57,864 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 4 docs from 0 to 5000 2021-05-02 23:10:57,865 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-02 23:10:57,865 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-02 23:10:57,865 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-02 23:10:57,867 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,868 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-02 23:10:57,868 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-02 23:10:57,869 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,869 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 12 docs from 500 to 980 2021-05-02 23:10:57,871 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 10 docs from 0 to 500 2021-05-02 23:10:57,871 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-02 23:10:57,871 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-02 23:10:57,871 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 500 to 980 2021-05-02 23:10:57,872 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 10 docs from 0 to 500 2021-05-02 23:10:57,872 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-02 23:10:57,874 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-02 23:10:57,875 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-02 23:10:57,877 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-02 23:10:57,877 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-02 23:10:57,878 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,878 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 6 docs from 500 to 980 2021-05-02 23:10:57,878 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-02 23:10:57,878 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-02 23:10:57,880 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-02 23:10:57,880 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-02 23:10:57,881 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:10:57,881 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-02 23:10:57,881 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-02 23:10:57,882 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-02 23:10:57,882 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-02 23:10:57,884 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-02 23:10:57,884 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-02 23:10:57,884 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-02 23:10:57,885 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,887 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-02 23:10:57,888 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-02 23:10:57,888 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,888 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-02 23:10:57,888 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 5 docs from 0 to 500 2021-05-02 23:10:57,888 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:10:57,890 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-02 23:10:57,891 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:10:57,891 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-02 23:10:57,891 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-02 23:10:57,891 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:10:57,892 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,892 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-02 23:10:57,894 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:10:57,894 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,894 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:10:57,895 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-02 23:10:57,895 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 3 docs from 0 to 500 2021-05-02 23:10:57,897 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-02 23:10:57,897 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:10:57,898 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:10:57,898 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:10:57,898 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,900 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:10:57,900 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:10:57,901 | test | ERROR | pool-5-thread-8 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:11:33,592 | test | INFO | pool-5-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****.**X | 32.282 | | 20.700 | *...*..*..X | 157.912 | | 135.721 | .....*X | 135.721 | | 103.215 | ...X | 103.215 | | 79.210 | ..*.X | 84.100 | | 68.295 | ..**...**.........*.........................*...................................X | 1569.032 | | 1547.132 | .........................................................................*..........................X | 2075.332 | | 1807.700 | ......................................................................................X | 1807.700 | | 1117.700 | ....................................................*X | 1117.700 | | 393.700 | .................*X | 393.700 | | 0.000 | *************************.**X | 31.192 | | 20.700 | X | 20.700 | | 10.300 | *X | 10.300 | | 0.000 | *************************************************.***X | 23.400 | | 14.399 | X | 14.399 | | 7.199 | *X | 7.199 | | 1.999 | *X | 1.999 | | 1.300 | X | 1.300 | | 0.900 | X | 0.900 | | 0.500 | **X | 0.500 | | 0.200 | X | 0.200 | | 0.100 | X | 0.100 | | 0.000 | *******X | 0.000 | +----------+-------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 23:11:35,434 | test | INFO | pool-5-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *********......*......*...........X | 605.306 | | 438.973 | ...............*X | 438.973 | | 271.273 | .........X | 271.273 | | 53.284 | .*...*....*..X | 295.722 | | 200.722 | ......*X | 200.722 | | 181.772 | .....*.....**..........*.........*.............X | 1131.300 | | 1007.099 | .....................................*X | 1007.099 | | 845.499 | ...............................X | 845.499 | | 676.156 | ........................*..**..X | 766.008 | | 704.886 | .........................*.X | 715.086 | | 692.926 | .........................X | 692.926 | | 639.891 | .......................*X | 639.891 | | 596.968 | .....................X | 596.968 | | 588.090 | .....................*.X | 627.608 | | 592.808 | .....................*X | 592.808 | | 571.718 | ....................X | 571.718 | | 513.645 | ..................X | 513.645 | | 493.714 | .................*X | 493.714 | | 412.668 | ..............*X | 412.668 | | 397.427 | ..............**........**....X | 730.445 | | 665.202 | ........................X | 665.202 | | 605.302 | ......................X | 605.302 | | 551.928 | ....................**X | 566.626 | | 563.383 | ....................*..**.........**..*.X | 922.107 | | 880.284 | ................................***...*..X | 1004.550 | | 979.613 | ....................................*X | 979.613 | | 913.114 | .................................**...**.....*.....*.............**..........*......*X | 2003.108 | | 1984.308 | ..........................................................................**........*.........*....*X | 2522.021 | | 2312.037 | .......................................................................................X | 2312.037 | | 2140.437 | ................................................................................*X | 2140.437 | | 2061.037 | .............................................................................**...X | 2142.439 | | 2134.239 | ................................................................................**....*..*..*X | 2349.664 | | 2248.005 | ....................................................................................*X | 2248.005 | | 2207.605 | ...................................................................................*.**........*......*.*X | 2623.357 | | 2221.457 | ...................................................................................*X | 2221.457 | | 1748.438 | .................................................................X | 1748.438 | | 1323.138 | .................................................X | 1323.138 | | 1204.223 | ............................................**X | 1204.223 | | 759.997 | ...........................**X | 759.997 | | 412.622 | ..............X | 412.622 | | 170.310 | .....X | 170.310 | | 34.310 | *X | 34.310 | | 0.000 | ***X | 0.000 | +----------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 23:11:36,602 | test | INFO | pool-5-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **************************************************************************************************.***X | 16.404 | | 14.599 | X | 14.599 | | 12.299 | X | 12.299 | | 10.101 | **X | 10.601 | | 7.401 | *X | 7.401 | | 5.600 | ***...*....**......*.....**....**....*.....**.........**........................***..............*..............*........*X | 1382.400 | +--------+-----------------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 23:11:36,661 | test | WARNING | Thread-8137 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:h-3-762000:QlrpPo6sig-RmTS_W-3-893000 create failed keys from task: [u'test_collections-744', u'test_collections-300', u'test_collections-48', u'test_collections-978', u'test_collections-117', u'test_collections-337', u'test_collections-578', u'test_collections-290', u'test_collections-1075', u'test_collections-1176', u'test_collections-1036'] 2021-05-02 23:11:36,661 | test | WARNING | Thread-8136 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:h-3-762000:eFKk43DYyzD3Uaz17Enn-3-892000 create failed keys from task: [u'test_collections-978', u'test_collections-117', u'test_collections-337', u'test_collections-744', u'test_collections-578', u'test_collections-300', u'test_collections-290', u'test_collections-1075', u'test_collections-1176', u'test_collections-1036'] 2021-05-02 23:11:36,684 | test | WARNING | Thread-8211 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:WCvq1dYATQr1P0bU15-3-233000:YP%baM-3-456000 create failed keys from task: [u'test_collections-978', u'test_collections-744', u'test_collections-62', u'test_collections-1176', u'test_collections-1036'] 2021-05-02 23:11:36,788 | test | WARNING | Thread-8329 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:2IMdZX4328EDgEpQWx-10-854000 update failed keys from task: [u'test_collections-913', u'test_collections-925', u'test_collections-931', u'test_collections-971'] 2021-05-02 23:11:36,789 | test | WARNING | Thread-8330 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F-10-853000 update failed keys from task: [u'test_collections-926', u'test_collections-905', u'test_collections-869', u'test_collections-962', u'test_collections-883', u'test_collections-782', u'test_collections-791'] 2021-05-02 23:11:36,789 | test | WARNING | Thread-8331 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 update failed keys from task: [u'test_collections-905', u'test_collections-635', u'test_collections-962', u'test_collections-883'] 2021-05-02 23:11:36,802 | test | WARNING | Thread-8360 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:B-2iMMFwyvH5GQsKdV-4-943000:NgAs-5-147000 create failed keys from task: [u'test_collections-639', u'test_collections-225'] 2021-05-02 23:11:36,803 | test | WARNING | Thread-8361 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:B-2iMMFwyvH5GQsKdV-4-943000:6fWtWOBPLReeW8IyX-5-146000 create failed keys from task: [u'test_collections-38', u'test_collections-729', u'test_collections-79', u'test_collections-1742', u'test_collections-244', u'test_collections-1349', u'test_collections-253', u'test_collections-1549', u'test_collections-1472', u'test_collections-1023'] 2021-05-02 23:11:36,805 | test | WARNING | Thread-8362 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:iJLzhAb5VclOEF53d-5-363000:S1NQn87aapfAiv_vtnK-5-538000 create failed keys from task: [u'test_collections-3072'] 2021-05-02 23:11:36,882 | test | WARNING | Thread-8385 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9nZOj4vIanO-10-865000:5C-10-865000 update failed keys from task: [u'test_collections-708', u'test_collections-709', u'test_collections-926', u'test_collections-546', u'test_collections-635', u'test_collections-962', u'test_collections-883', u'test_collections-791'] 2021-05-02 23:11:36,788 | test | WARNING | Thread-8328 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Jx4QNINElmqCAMNdF-10-853000 update failed keys from task: [u'test_collections-956'] 2021-05-02 23:11:36,805 | test | WARNING | Thread-8363 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:iJLzhAb5VclOEF53d-5-363000:1hhKWENCHXJJfmC-5-537000 create failed keys from task: [u'test_collections-1665', u'test_collections-2613', u'test_collections-2615', u'test_collections-1946', u'test_collections-2075', u'test_collections-2492', u'test_collections-2050', u'test_collections-2502', u'test_collections-1657', u'test_collections-2068', u'test_collections-2783', u'test_collections-2785'] 2021-05-02 23:11:38,434 | test | WARNING | Thread-8213 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ihvs5Co6fc_N%-10-842000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,434 | test | WARNING | Thread-8212 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:WCvq1dYATQr1P0bU15-3-233000:E-3-457000 create failed keys from task: [u'test_collections-978', u'test_collections-744', u'test_collections-107', u'test_collections-1075', u'test_collections-1176', u'test_collections-1036'] 2021-05-02 23:11:38,434 | test | WARNING | Thread-8402 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:YyH70SSha2-4-477000:9r8y-4-677000 create failed keys from task: [u'test_collections-714', u'test_collections-223', u'test_collections-684', u'test_collections-172'] 2021-05-02 23:11:38,436 | test | WARNING | Thread-8216 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ru8z8PvywzIbxa-10-840000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,434 | test | WARNING | Thread-8387 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 update failed keys from task: [u'test_collections-829', u'test_collections-905', u'test_collections-748', u'test_collections-922', u'test_collections-846', u'test_collections-546', u'test_collections-503', u'test_collections-561', u'test_collections-878', u'test_collections-637', u'test_collections-932', u'test_collections-793', u'test_collections-791'] 2021-05-02 23:11:38,437 | test | WARNING | Thread-8214 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:Eq%PnX6J-10-839000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,443 | test | WARNING | Thread-8215 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:s%ZD-10-841000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,428 | test | WARNING | Thread-8329 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:2IMdZX4328EDgEpQWx-10-854000 delete failed keys from task: [u'test_collections-234', u'test_collections-493'] 2021-05-02 23:11:38,450 | test | WARNING | Thread-8385 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9nZOj4vIanO-10-865000:5C-10-865000 delete failed keys from task: [u'test_collections-1', u'test_collections-216', u'test_collections-468', u'test_collections-147', u'test_collections-213', u'test_collections-386', u'test_collections-252', u'test_collections-383', u'test_collections-271'] 2021-05-02 23:11:38,450 | test | WARNING | Thread-8331 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 delete failed keys from task: [u'test_collections-1', u'test_collections-439', u'test_collections-456', u'test_collections-468', u'test_collections-234', u'test_collections-147', u'test_collections-213', u'test_collections-383', u'test_collections-271'] 2021-05-02 23:11:38,470 | test | WARNING | Thread-8215 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:s%ZD-10-841000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,470 | test | WARNING | Thread-8214 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:Eq%PnX6J-10-839000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,471 | test | WARNING | Thread-8213 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ihvs5Co6fc_N%-10-842000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,471 | test | WARNING | Thread-8428 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:ewQTD-10-855000 delete failed keys from task: [u'test_collections-35'] 2021-05-02 23:11:38,471 | test | WARNING | Thread-8216 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:Ru8z8PvywzIbxa-10-840000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,480 | test | WARNING | Thread-8428 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:ewQTD-10-855000 create failed keys from task: [u'test_collections-2661'] 2021-05-02 23:11:38,482 | test | WARNING | Thread-8331 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:fmIDYu3%4wXXPF1sahMI-10-852000 create failed keys from task: [u'test_collections-2855'] 2021-05-02 23:11:38,489 | test | WARNING | Thread-8330 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F-10-853000 delete failed keys from task: [u'test_collections-1', u'test_collections-456', u'test_collections-468', u'test_collections-234', u'test_collections-213', u'test_collections-243', u'test_collections-252', u'test_collections-383', u'test_collections-271'] 2021-05-02 23:11:38,553 | test | WARNING | Thread-8405 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:KL3O%n97isrbjno-10-858000 update failed keys from task: [u'test_collections-906', u'test_collections-647', u'test_collections-522', u'test_collections-567', u'test_collections-916', u'test_collections-517', u'test_collections-716', u'test_collections-537', u'test_collections-533', u'test_collections-895', u'test_collections-552'] 2021-05-02 23:11:38,555 | test | WARNING | Thread-8403 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:YyH70SSha2-4-477000:75hRg7m-4-676000 create failed keys from task: [u'test_collections-729', u'test_collections-223', u'test_collections-1434', u'test_collections-166', u'test_collections-121', u'test_collections-5', u'test_collections-1432'] 2021-05-02 23:11:38,558 | test | WARNING | Thread-8406 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:6M-10-857000 update failed keys from task: [u'test_collections-906', u'test_collections-916', u'test_collections-517', u'test_collections-716', u'test_collections-647', u'test_collections-537', u'test_collections-533', u'test_collections-567', u'test_collections-895', u'test_collections-552'] 2021-05-02 23:11:38,559 | test | WARNING | Thread-8386 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 update failed keys from task: [u'test_collections-709', u'test_collections-922', u'test_collections-546', u'test_collections-635', u'test_collections-932', u'test_collections-883', u'test_collections-791'] 2021-05-02 23:11:38,565 | test | WARNING | Thread-8330 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F-10-853000 create failed keys from task: [u'test_collections-2734', u'test_collections-2837'] 2021-05-02 23:11:38,588 | test | WARNING | Thread-8251 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:nGItBB8n1DXPDC-10-829000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,614 | test | WARNING | Thread-8251 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:nGItBB8n1DXPDC-10-829000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,641 | test | WARNING | Thread-8252 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:A5D%iy0zrcSp3tEwEAU-10-830000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,661 | test | WARNING | Thread-8387 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 delete failed keys from task: [u'test_collections-349', u'test_collections-203', u'test_collections-324', u'test_collections-386', u'test_collections-485', u'test_collections-145', u'test_collections-216', u'test_collections-413', u'test_collections-253', u'test_collections-231', u'test_collections-393'] 2021-05-02 23:11:38,663 | test | WARNING | Thread-8386 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9nZOj4vIanO-10-865000:zLuYLf2aPNCxR-10-871000 delete failed keys from task: [u'test_collections-1', u'test_collections-216', u'test_collections-468', u'test_collections-147', u'test_collections-213', u'test_collections-386', u'test_collections-252', u'test_collections-383', u'test_collections-271'] 2021-05-02 23:11:38,667 | test | WARNING | Thread-8252 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:A5D%iy0zrcSp3tEwEAU-10-830000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,717 | test | WARNING | Thread-8405 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:KL3O%n97isrbjno-10-858000 delete failed keys from task: [u'test_collections-428', u'test_collections-248', u'test_collections-135', u'test_collections-267', u'test_collections-422', u'test_collections-496', u'test_collections-497', u'test_collections-240'] 2021-05-02 23:11:38,750 | test | WARNING | Thread-8406 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:6M-10-857000 delete failed keys from task: [u'test_collections-428', u'test_collections-135', u'test_collections-267', u'test_collections-422', u'test_collections-496', u'test_collections-222', u'test_collections-497', u'test_collections-240'] 2021-05-02 23:11:38,755 | test | WARNING | Thread-8387 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9nZOj4vIanO-10-865000:GSGmuFs8kLkIL2soz58v-10-868000 create failed keys from task: [u'test_collections-2816', u'test_collections-2726'] 2021-05-02 23:11:38,796 | test | WARNING | Thread-8424 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 update failed keys from task: [u'test_collections-788', u'test_collections-681'] 2021-05-02 23:11:38,802 | test | WARNING | Thread-8425 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:9FEgAfFzv_8XkE30U-10-856000 update failed keys from task: [u'test_collections-510'] 2021-05-02 23:11:38,816 | test | WARNING | Thread-8405 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:KL3O%n97isrbjno-10-858000 create failed keys from task: [u'test_collections-2765'] 2021-05-02 23:11:38,818 | test | WARNING | Thread-8425 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:9FEgAfFzv_8XkE30U-10-856000 delete failed keys from task: [u'test_collections-42'] 2021-05-02 23:11:38,829 | test | WARNING | Thread-8217 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:yO4bAhC3Tkv7--10-841000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,832 | test | WARNING | Thread-8218 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:wIq5FBdocmK-10-839000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-02 23:11:38,835 | test | WARNING | Thread-8424 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 delete failed keys from task: [u'test_collections-356', u'test_collections-332'] 2021-05-02 23:11:38,836 | test | WARNING | Thread-8425 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:9FEgAfFzv_8XkE30U-10-856000 create failed keys from task: [u'test_collections-2653'] 2021-05-02 23:11:38,845 | test | WARNING | Thread-8407 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:3VW5-10-857000 update failed keys from task: [u'test_collections-979', u'test_collections-713', u'test_collections-824', u'test_collections-557', u'test_collections-832', u'test_collections-833', u'test_collections-756', u'test_collections-886', u'test_collections-641', u'test_collections-532'] 2021-05-02 23:11:38,846 | test | WARNING | Thread-8409 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:kRpj-10-858000 update failed keys from task: [u'test_collections-713', u'test_collections-824', u'test_collections-557', u'test_collections-832', u'test_collections-833', u'test_collections-756', u'test_collections-886', u'test_collections-641', u'test_collections-532'] 2021-05-02 23:11:38,846 | test | WARNING | Thread-8408 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:CqtEagom7q-10-858000 update failed keys from task: [u'test_collections-647', u'test_collections-522', u'test_collections-567', u'test_collections-838', u'test_collections-916', u'test_collections-517', u'test_collections-859', u'test_collections-537', u'test_collections-533', u'test_collections-895', u'test_collections-892', u'test_collections-552'] 2021-05-02 23:11:38,859 | test | WARNING | Thread-8218 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:wIq5FBdocmK-10-839000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,862 | test | WARNING | Thread-8217 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:RiFZyaWh1xlhtBTsC-10-837000:yO4bAhC3Tkv7--10-841000 delete failed keys from task: [u'test_collections-2'] 2021-05-02 23:11:38,865 | test | WARNING | Thread-8406 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:6M-10-857000 create failed keys from task: [u'test_collections-2874'] 2021-05-02 23:11:38,867 | test | WARNING | Thread-8424 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NgtA%HwWJ01Lp7H8RiD-10-859000 create failed keys from task: [u'test_collections-2501', u'test_collections-2891'] 2021-05-02 23:11:38,887 | test | WARNING | Thread-8413 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:q-10-855000 update failed keys from task: [u'test_collections-765', u'test_collections-674', u'test_collections-697'] 2021-05-02 23:11:38,947 | test | WARNING | Thread-8413 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:q-10-855000 delete failed keys from task: [u'test_collections-219', u'test_collections-106', u'test_collections-95', u'test_collections-426', u'test_collections-459', u'test_collections-226', u'test_collections-389', u'test_collections-431', u'test_collections-492'] 2021-05-02 23:11:38,974 | test | WARNING | Thread-8419 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:KyAXYneDVJMbur-10-855000 update failed keys from task: [u'test_collections-658', u'test_collections-579', u'test_collections-630'] 2021-05-02 23:11:38,976 | test | WARNING | Thread-8250 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:NaVJU7HPZhXvKjtkT-4-126000:IH23P8GSOh--4-285000 create failed keys from task: [u'test_collections-978', u'test_collections-337', u'test_collections-744', u'test_collections-126', u'test_collections-578', u'test_collections-300', u'test_collections-290', u'test_collections-1176', u'test_collections-1036'] 2021-05-02 23:11:38,977 | test | WARNING | Thread-8249 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:NaVJU7HPZhXvKjtkT-4-126000:RK%Zkse0O4cr2wms6Jf-4-281000 create failed keys from task: [u'test_collections-744', u'test_collections-126', u'test_collections-300', u'test_collections-162', u'test_collections-978', u'test_collections-337', u'test_collections-578', u'test_collections-290', u'test_collections-1075', u'test_collections-1176', u'test_collections-1036'] 2021-05-02 23:11:38,980 | test | WARNING | Thread-8421 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:UeTdD-10-856000 update failed keys from task: [u'test_collections-590'] 2021-05-02 23:11:38,983 | test | WARNING | Thread-8420 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:Z-10-856000 update failed keys from task: [u'test_collections-658', u'test_collections-669', u'test_collections-853', u'test_collections-663'] 2021-05-02 23:11:39,005 | test | WARNING | Thread-8421 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:UeTdD-10-856000 delete failed keys from task: [u'test_collections-75', u'test_collections-179', u'test_collections-355', u'test_collections-268', u'test_collections-364', u'test_collections-82', u'test_collections-374'] 2021-05-02 23:11:39,006 | test | WARNING | Thread-8409 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:kRpj-10-858000 delete failed keys from task: [u'test_collections-108', u'test_collections-128', u'test_collections-115', u'test_collections-346', u'test_collections-434', u'test_collections-259', u'test_collections-466', u'test_collections-133', u'test_collections-274', u'test_collections-350'] 2021-05-02 23:11:39,036 | test | WARNING | Thread-8420 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:Z-10-856000 delete failed keys from task: [u'test_collections-419', u'test_collections-355', u'test_collections-111', u'test_collections-364'] 2021-05-02 23:11:39,039 | test | WARNING | Thread-8413 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:q-10-855000 create failed keys from task: [u'test_collections-2535', u'test_collections-2684'] 2021-05-02 23:11:39,045 | test | WARNING | Thread-8419 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:KyAXYneDVJMbur-10-855000 delete failed keys from task: [u'test_collections-17', u'test_collections-61', u'test_collections-178', u'test_collections-142', u'test_collections-372'] 2021-05-02 23:11:39,048 | test | WARNING | Thread-8420 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:Z-10-856000 create failed keys from task: [u'test_collections-2551', u'test_collections-2664'] 2021-05-02 23:11:39,061 | test | WARNING | Thread-8412 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:lL1HnFn-10-859000 update failed keys from task: [u'test_collections-609', u'test_collections-827', u'test_collections-778', u'test_collections-574', u'test_collections-891', u'test_collections-690'] 2021-05-02 23:11:39,062 | test | WARNING | Thread-8419 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:KyAXYneDVJMbur-10-855000 create failed keys from task: [u'test_collections-2589', u'test_collections-2746'] 2021-05-02 23:11:39,065 | test | WARNING | Thread-8410 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 update failed keys from task: [u'test_collections-509', u'test_collections-707', u'test_collections-944', u'test_collections-862', u'test_collections-540'] 2021-05-02 23:11:39,068 | test | WARNING | Thread-8408 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:CqtEagom7q-10-858000 delete failed keys from task: [u'test_collections-425', u'test_collections-96', u'test_collections-225', u'test_collections-434', u'test_collections-422', u'test_collections-499', u'test_collections-221', u'test_collections-131', u'test_collections-165', u'test_collections-341'] 2021-05-02 23:11:39,071 | test | WARNING | Thread-8415 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:RWY022TdyYwK4GIe2ShY-5-752000 create failed keys from task: [u'test_collections-978'] 2021-05-02 23:11:39,092 | test | WARNING | Thread-8407 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:3VW5-10-857000 delete failed keys from task: [u'test_collections-108', u'test_collections-128', u'test_collections-346', u'test_collections-434', u'test_collections-259', u'test_collections-466', u'test_collections-133', u'test_collections-274', u'test_collections-350'] 2021-05-02 23:11:39,098 | test | WARNING | Thread-8421 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:UeTdD-10-856000 create failed keys from task: [u'test_collections-2786'] 2021-05-02 23:11:39,174 | test | WARNING | Thread-8411 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 update failed keys from task: [u'test_collections-509', u'test_collections-707', u'test_collections-944', u'test_collections-862', u'test_collections-540'] 2021-05-02 23:11:39,176 | test | WARNING | Thread-8427 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NORvTSE7XgYIZeJOov-10-858000 delete failed keys from task: [u'test_collections-70'] 2021-05-02 23:11:39,180 | test | WARNING | Thread-8410 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 delete failed keys from task: [u'test_collections-418', u'test_collections-33', u'test_collections-11', u'test_collections-138', u'test_collections-300', u'test_collections-430', u'test_collections-474', u'test_collections-290'] 2021-05-02 23:11:39,187 | test | WARNING | Thread-8417 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collection_1 update failed keys from task: [u'test_collections-945', u'test_collections-502', u'test_collections-821', u'test_collections-843'] 2021-05-02 23:11:39,188 | test | WARNING | Thread-8422 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:H4p-5kk-10-859000 update failed keys from task: [u'test_collections-928', u'test_collections-776', u'test_collections-754', u'test_collections-586', u'test_collections-596'] 2021-05-02 23:11:39,191 | test | WARNING | Thread-8416 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:-B0gYMS-10-856000 update failed keys from task: [u'test_collections-808', u'test_collections-765'] 2021-05-02 23:11:39,193 | test | WARNING | Thread-8418 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:vx1gX_7qwnoKAOF-10-855000 update failed keys from task: [u'test_collections-658', u'test_collections-579', u'test_collections-630'] 2021-05-02 23:11:39,194 | test | WARNING | Thread-8423 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:2H3a3w53MQrlPaQTR4S-10-859000 update failed keys from task: [u'test_collections-871'] 2021-05-02 23:11:39,196 | test | WARNING | Thread-8414 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:dlKrjx-10-856000 update failed keys from task: [u'test_collections-627', u'test_collections-969', u'test_collections-702', u'test_collections-865', u'test_collections-820', u'test_collections-765', u'test_collections-697'] 2021-05-02 23:11:39,197 | test | WARNING | Thread-8412 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:lL1HnFn-10-859000 delete failed keys from task: [u'test_collections-418', u'test_collections-11', u'test_collections-33', u'test_collections-258', u'test_collections-430', u'test_collections-474', u'test_collections-450', u'test_collections-182'] 2021-05-02 23:11:39,203 | test | WARNING | Thread-8410 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:1a5W5gD9LKvikXMx%VDT-10-855000 create failed keys from task: [u'test_collections-2517', u'test_collections-2573', u'test_collections-2542'] 2021-05-02 23:11:39,207 | test | WARNING | Thread-8423 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:2H3a3w53MQrlPaQTR4S-10-859000 delete failed keys from task: [u'test_collections-25', u'test_collections-47', u'test_collections-262'] 2021-05-02 23:11:39,216 | test | WARNING | Thread-8417 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collection_1 delete failed keys from task: [u'test_collections-337', u'test_collections-462'] 2021-05-02 23:11:39,219 | test | WARNING | Thread-8416 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:-B0gYMS-10-856000 delete failed keys from task: [u'test_collections-54', u'test_collections-76', u'test_collections-94', u'test_collections-73', u'test_collections-343', u'test_collections-354', u'test_collections-187', u'test_collections-270'] 2021-05-02 23:11:39,221 | test | WARNING | Thread-8417 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collection_1 create failed keys from task: [u'test_collections-2826'] 2021-05-02 23:11:39,223 | test | WARNING | Thread-8418 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:vx1gX_7qwnoKAOF-10-855000 delete failed keys from task: [u'test_collections-61', u'test_collections-178', u'test_collections-142', u'test_collections-372'] 2021-05-02 23:11:39,230 | test | WARNING | Thread-8418 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:vx1gX_7qwnoKAOF-10-855000 create failed keys from task: [u'test_collections-2570'] 2021-05-02 23:11:39,253 | test | WARNING | Thread-8411 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 delete failed keys from task: [u'test_collections-418', u'test_collections-33', u'test_collections-11', u'test_collections-138', u'test_collections-300', u'test_collections-430', u'test_collections-474', u'test_collections-290'] 2021-05-02 23:11:39,256 | test | WARNING | Thread-8412 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:lL1HnFn-10-859000 create failed keys from task: [u'test_collections-2600', u'test_collections-2622', u'test_collections-2788'] 2021-05-02 23:11:39,267 | test | WARNING | Thread-8411 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NHvJcH4Sewl6bLrm7me-10-855000 create failed keys from task: [u'test_collections-2573'] 2021-05-02 23:11:39,285 | test | WARNING | Thread-8422 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:H4p-5kk-10-859000 delete failed keys from task: [u'test_collections-471'] 2021-05-02 23:11:39,286 | test | WARNING | Thread-8422 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:H4p-5kk-10-859000 create failed keys from task: [u'test_collections-2569'] 2021-05-02 23:11:39,305 | test | WARNING | Thread-8414 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:dlKrjx-10-856000 delete failed keys from task: [u'test_collections-54', u'test_collections-76', u'test_collections-94', u'test_collections-73', u'test_collections-343', u'test_collections-354', u'test_collections-453', u'test_collections-187', u'test_collections-270'] 2021-05-02 23:11:48,862 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 23:12:49,275 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-02 23:12:49,276 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-02 23:13:06,917 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-02 23:13:06,918 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-02 23:13:29,539 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-02 23:13:29,540 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-02 23:14:14,117 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-02 23:14:32,138 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1579999924 seconds to complete 2021-05-02 23:14:32,157 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1900.723s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_3 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_4', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 4, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-02 23:14:42,200 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-02 23:14:43,262 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-02 23:14:43,828 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116750 type=client open_count=1 channel=[id: 0x35e08c2d, 0.0.0.0/0.0.0.0:54477] timeout=300.0>) 2021-05-02 23:14:49,467 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116751 type=client open_count=1 channel=[id: 0xf689bc48, 0.0.0.0/0.0.0.0:43093] timeout=300.0>) 2021-05-02 23:14:51,505 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116752 type=client open_count=1 channel=[id: 0x8119699a, 0.0.0.0/0.0.0.0:39163] timeout=300.0>) 2021-05-02 23:14:53,529 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116753 type=client open_count=1 channel=[id: 0xad156b9d, 0.0.0.0/0.0.0.0:38668] timeout=300.0>) 2021-05-02 23:14:55,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116754 type=client open_count=1 channel=[id: 0x36b0e1df, 0.0.0.0/0.0.0.0:43477] timeout=300.0>) 2021-05-02 23:14:57,661 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116755 type=client open_count=1 channel=[id: 0x9b93af79, 0.0.0.0/0.0.0.0:56799] timeout=300.0>) 2021-05-02 23:14:59,683 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116756 type=client open_count=1 channel=[id: 0x459ae031, 0.0.0.0/0.0.0.0:43850] timeout=300.0>) 2021-05-02 23:15:01,704 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116757 type=client open_count=1 channel=[id: 0xfde283da, 0.0.0.0/0.0.0.0:35097] timeout=300.0>) 2021-05-02 23:15:09,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116758 type=client open_count=1 channel=[id: 0xf8abc51c, 0.0.0.0/0.0.0.0:47839] timeout=300.0>) 2021-05-02 23:15:11,267 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116759 type=client open_count=1 channel=[id: 0xc9afeb0b, 0.0.0.0/0.0.0.0:52918] timeout=300.0>) 2021-05-02 23:15:13,289 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x11675a type=client open_count=1 channel=[id: 0xdb9fad68, 0.0.0.0/0.0.0.0:43609] timeout=300.0>) 2021-05-02 23:15:15,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x11675b type=client open_count=1 channel=[id: 0x4990f36e, 0.0.0.0/0.0.0.0:41837] timeout=300.0>) 2021-05-02 23:15:17,331 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x11675c type=client open_count=1 channel=[id: 0xd1964784, 0.0.0.0/0.0.0.0:40899] timeout=300.0>) 2021-05-02 23:15:19,354 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x11675d type=client open_count=1 channel=[id: 0x57fe64fc, 0.0.0.0/0.0.0.0:42419] timeout=300.0>) 2021-05-02 23:15:21,377 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x11675e type=client open_count=1 channel=[id: 0x67eacd86, 0.0.0.0/0.0.0.0:39745] timeout=300.0>) 2021-05-02 23:15:23,400 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x11675f type=client open_count=1 channel=[id: 0x59457033, 0.0.0.0/0.0.0.0:49729] timeout=300.0>) 2021-05-02 23:15:25,423 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116760 type=client open_count=1 channel=[id: 0x2313267b, 0.0.0.0/0.0.0.0:34945] timeout=300.0>) 2021-05-02 23:15:32,980 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116761 type=client open_count=1 channel=[id: 0x14178e81, 0.0.0.0/0.0.0.0:60947] timeout=300.0>) 2021-05-02 23:15:35,003 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116762 type=client open_count=1 channel=[id: 0x83a0c952, 0.0.0.0/0.0.0.0:60469] timeout=300.0>) 2021-05-02 23:15:37,029 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116763 type=client open_count=1 channel=[id: 0xdc431dfe, 0.0.0.0/0.0.0.0:51787] timeout=300.0>) 2021-05-02 23:15:39,049 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116764 type=client open_count=1 channel=[id: 0x02cc3d4d, 0.0.0.0/0.0.0.0:40763] timeout=300.0>) 2021-05-02 23:15:41,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116765 type=client open_count=1 channel=[id: 0xa500f1ae, 0.0.0.0/0.0.0.0:34409] timeout=300.0>) 2021-05-02 23:15:43,095 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116766 type=client open_count=1 channel=[id: 0x93257b17, 0.0.0.0/0.0.0.0:36193] timeout=300.0>) 2021-05-02 23:15:45,118 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116767 type=client open_count=1 channel=[id: 0x02be2e0e, 0.0.0.0/0.0.0.0:57159] timeout=300.0>) 2021-05-02 23:15:47,141 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116768 type=client open_count=1 channel=[id: 0x0c62e538, 0.0.0.0/0.0.0.0:46817] timeout=300.0>) 2021-05-02 23:15:49,167 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116769 type=client open_count=1 channel=[id: 0x67e42717, 0.0.0.0/0.0.0.0:60173] timeout=300.0>) 2021-05-02 23:15:51,188 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x11676a type=client open_count=1 channel=[id: 0x876c92f3, 0.0.0.0/0.0.0.0:59448] timeout=300.0>) 2021-05-02 23:15:59,288 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x11676b type=client open_count=1 channel=[id: 0x1ad23f8e, 0.0.0.0/0.0.0.0:42415] timeout=300.0>) 2021-05-02 23:16:01,309 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x11676c type=client open_count=1 channel=[id: 0x1aa503c8, 0.0.0.0/0.0.0.0:33230] timeout=300.0>) 2021-05-02 23:16:03,332 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x11676d type=client open_count=1 channel=[id: 0x7eb7f8c8, 0.0.0.0/0.0.0.0:36133] timeout=300.0>) 2021-05-02 23:16:05,355 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x11676e type=client open_count=1 channel=[id: 0xbcfc5cb4, 0.0.0.0/0.0.0.0:47547] timeout=300.0>) 2021-05-02 23:16:07,378 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x11676f type=client open_count=1 channel=[id: 0x06c0215f, 0.0.0.0/0.0.0.0:49247] timeout=300.0>) 2021-05-02 23:16:09,400 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116770 type=client open_count=1 channel=[id: 0x51651a81, 0.0.0.0/0.0.0.0:56502] timeout=300.0>) 2021-05-02 23:16:11,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116771 type=client open_count=1 channel=[id: 0x542d486f, 0.0.0.0/0.0.0.0:55514] timeout=300.0>) 2021-05-02 23:16:13,441 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116772 type=client open_count=1 channel=[id: 0xa02c8a6f, 0.0.0.0/0.0.0.0:53226] timeout=300.0>) 2021-05-02 23:16:15,463 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116773 type=client open_count=1 channel=[id: 0xda198a1b, 0.0.0.0/0.0.0.0:46103] timeout=300.0>) 2021-05-02 23:16:20,190 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116774 type=client open_count=1 channel=[id: 0xeb08b17c, 0.0.0.0/0.0.0.0:44933] timeout=300.0>) 2021-05-02 23:16:22,213 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116775 type=client open_count=1 channel=[id: 0xa64a3772, 0.0.0.0/0.0.0.0:55919] timeout=300.0>) 2021-05-02 23:16:24,239 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116776 type=client open_count=1 channel=[id: 0xcbde39a6, 0.0.0.0/0.0.0.0:46841] timeout=300.0>) 2021-05-02 23:16:26,262 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116777 type=client open_count=1 channel=[id: 0x13aaf6c3, 0.0.0.0/0.0.0.0:38425] timeout=300.0>) 2021-05-02 23:16:28,283 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116778 type=client open_count=1 channel=[id: 0x06b9ba84, 0.0.0.0/0.0.0.0:42653] timeout=300.0>) 2021-05-02 23:16:30,305 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 23:16:47,137 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-02 23:17:01,490 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-02 23:17:01,519 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-02 23:17:01,579 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-02 23:17:34,210 | test | INFO | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.76426047451 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-02 23:17:34,240 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:17:39,308 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:17:49,398 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-02 23:17:49,434 | test | INFO | pool-6-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.2249999046 sec 2021-05-02 23:17:49,822 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-02 23:18:30,085 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-02 23:18:30,404 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 629410512 | 30531766 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 531399840 | 106931261 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 609998640 | 77788993 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-02 23:18:30,405 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-02 23:19:22,430 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 23:19:22,447 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 23:35:42,490 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *.**...*.....*X | 3.000 | | 2.400 | ......X | 2.400 | | 1.600 | ...X | 1.600 | | 0.000 | ****************************************************************************************************..*..X | 1.500 | | 1.200 | ..*X | 1.200 | | 0.800 | .*X | 0.800 | | 0.000 | ********...*........*..*X | 4.500 | | 3.699 | ..........X | 3.699 | | 2.100 | .....****....*.........***X | 6.501 | | 5.300 | ...............X | 5.300 | | 1.300 | ..*X | 1.300 | | 0.400 | X | 0.400 | | 0.000 | *****.......*...............**.....*X | 9.005 | | 6.603 | ...................X | 6.603 | | 1.800 | ....X | 1.800 | | 0.000 | **.*..**.*X | 1.500 | | 1.100 | ..X | 1.100 | | 1.100 | ..**..**......*X | 3.700 | | 3.100 | ........*X | 3.100 | | 2.300 | ......***....**......*........X | 8.198 | | 6.699 | ...................*X | 6.699 | | 4.699 | .............X | 4.699 | | 1.500 | ...*X | 1.500 | | 1.200 | ..*X | 1.200 | | 0.800 | .X | 0.800 | | 0.000 | ***..*.....**..*X | 3.000 | | 2.200 | .....X | 2.200 | | 0.600 | X | 0.600 | | 0.599 | ***........................**..*X | 9.101 | | 8.701 | .........................X | 8.701 | | 7.101 | ....................*X | 7.101 | | 6.800 | ...................*X | 6.800 | | 4.800 | .............X | 4.800 | | 1.200 | ..X | 1.200 | | 0.000 | *******.*...*X | 1.400 | | 1.300 | ..*X | 1.300 | | 0.100 | **X | 0.100 | | 0.000 | *...*........**........**....*.......*X | 9.801 | | 7.100 | ....................*X | 7.100 | | 5.100 | ..............X | 5.100 | | 1.200 | ..**.X | 1.500 | | 1.100 | ..X | 1.100 | | 0.300 | ****...*.....**...**........*...............*X | 11.099 | | 10.799 | ................................X | 10.799 | | 7.596 | ......................X | 7.596 | | 2.801 | .......*X | 2.801 | | 2.201 | .....**....X | 3.401 | | 2.600 | ......*X | 2.600 | | 2.300 | ......*..*..........**....*X | 7.801 | | 6.201 | ..................X | 6.201 | | 3.001 | ........*X | 3.001 | | 1.200 | ..*X | 1.200 | | 0.000 | **************....*.......*.*X | 4.200 | | 3.000 | ........X | 3.000 | | 0.600 | **..**...................**X | 7.202 | | 6.602 | ...................X | 6.602 | | 5.000 | ..............X | 5.000 | | 1.800 | ....*X | 1.800 | | 1.200 | ..*X | 1.200 | | 0.000 | *******...**...................**X | 7.201 | | 6.601 | ...................X | 6.601 | | 6.199 | ..................X | 6.199 | | 5.398 | ...............**.*X | 5.698 | | 4.100 | ...........X | 4.100 | | 0.900 | .X | 0.900 | | 0.000 | ********..**.*X | 1.100 | | 0.700 | .**...**.........**...........................*...*X | 14.402 | | 13.201 | .......................................X | 13.201 | | 6.400 | ..................*X | 6.400 | | 4.300 | ............*X | 4.300 | | 2.300 | ......**X | 2.300 | | 2.300 | ......*.........*................................**...............*X | 20.690 | | 11.102 | .................................*X | 11.102 | | 6.602 | ...................*X | 6.602 | | 5.001 | ..............**.*.*X | 5.701 | | 4.101 | ...........X | 4.101 | | 0.899 | .*X | 0.899 | | 0.000 | **.....***.......*........*......*X | 8.599 | | 6.799 | ...................*X | 6.799 | | 6.000 | .................*.*...*X | 7.198 | | 5.998 | .................*X | 5.998 | | 3.600 | ..........X | 3.600 | | 3.001 | ........*X | 3.001 | | 2.201 | .....X | 2.201 | | 0.600 | *X | 0.600 | | 0.600 | **........*..............*........**..............**...........................................**............X | 32.608 | | 29.408 | .........................................................................................X | 29.408 | | 14.198 | ..........................................*X | 14.198 | | 7.598 | ......................**.*................*....................**.**...X | 21.003 | | 14.703 | ............................................*X | 14.703 | | 9.900 | .............................X | 9.900 | | 0.300 | *X | 0.300 | | 0.000 | ********************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-05-02 23:35:44,206 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....**......................**.................................................*.........................*X | 1249.300 | | 1075.600 | .....................................................................................*X | 1075.600 | | 908.400 | .......................................................................*X | 908.400 | | 856.100 | ...................................................................X | 856.100 | | 745.200 | ..........................................................X | 745.200 | | 725.700 | .........................................................***......*.....**...X | 902.403 | | 885.165 | .....................................................................*X | 885.165 | | 853.503 | ...................................................................*X | 853.503 | | 808.830 | ...............................................................X | 808.830 | | 747.930 | ..........................................................X | 747.930 | | 713.627 | ........................................................**X | 713.627 | | 536.000 | .........................................*X | 536.000 | | 454.800 | ...................................*X | 454.800 | | 419.838 | ................................X | 419.838 | | 376.838 | .............................**.**...**.*....*..**......***..*.*.*X | 644.300 | | 625.000 | .................................................*X | 625.000 | | 586.300 | .............................................X | 586.300 | | 579.677 | .............................................*..**...*.*X | 656.464 | | 655.716 | ...................................................*X | 655.716 | | 616.116 | ................................................*X | 616.116 | | 599.206 | ..............................................*X | 599.206 | | 584.790 | .............................................*.**.*X | 606.554 | | 515.435 | ........................................*X | 515.435 | | 449.135 | ..................................*X | 449.135 | | 439.042 | ..................................X | 439.042 | | 409.442 | ...............................**..**.*.***..*...*****...**..**.*.X | 612.116 | | 611.616 | ...............................................**.*.*X | 632.990 | | 632.190 | .................................................*X | 632.190 | | 587.240 | ..............................................X | 587.240 | | 510.818 | .......................................*X | 510.818 | | 500.235 | .......................................*.*..**X | 537.206 | | 501.652 | .......................................X | 501.652 | | 476.652 | .....................................X | 476.652 | | 444.305 | ..................................**..**.*....*.**..***.*X | 576.436 | | 569.836 | ............................................*..***X | 591.956 | | 576.741 | .............................................X | 576.741 | | 511.222 | .......................................X | 511.222 | | 466.222 | ....................................*X | 466.222 | | 438.228 | ..................................X | 438.228 | | 388.490 | ..............................****.*....****.X | 470.330 | | 445.987 | ..................................X | 445.987 | | 440.387 | ..................................*X | 440.387 | | 428.921 | .................................*.**..**...*...***X | 547.062 | | 527.062 | .........................................X | 527.062 | | 500.021 | .......................................X | 500.021 | | 478.621 | .....................................*X | 478.621 | | 468.421 | ....................................X | 468.421 | | 456.500 | ...................................**.*X | 471.200 | | 469.100 | ....................................X | 469.100 | | 457.100 | ...................................*X | 457.100 | | 444.800 | ..................................*X | 444.800 | | 421.700 | ................................*X | 421.700 | | 405.043 | ...............................*X | 405.043 | | 402.743 | ...............................*.**..**......***.*..*.*X | 574.674 | | 557.274 | ...........................................*X | 557.274 | | 536.000 | .........................................X | 536.000 | | 511.700 | .......................................**.**.***X | 535.276 | | 519.376 | ........................................X | 519.376 | | 510.576 | .......................................*X | 510.576 | | 459.378 | ...................................*X | 459.378 | | 442.378 | ..................................X | 442.378 | | 377.578 | .............................*..**.*...****...*.......***X | 586.040 | | 569.840 | ............................................***...X | 600.417 | | 585.417 | .............................................*..*X | 601.438 | | 579.892 | .............................................**X | 581.992 | | 532.720 | .........................................X | 532.720 | | 478.967 | .....................................*X | 478.967 | | 437.867 | ..................................X | 437.867 | | 403.518 | ...............................**..**....*....**.*X | 547.084 | | 515.741 | ........................................X | 515.741 | | 494.793 | ......................................X | 494.793 | | 478.693 | .....................................***...**.**.*.*.*X | 565.757 | | 559.245 | ...........................................X | 559.245 | | 526.638 | .........................................*X | 526.638 | | 491.938 | ......................................*X | 491.938 | | 472.938 | ....................................X | 472.938 | | 429.600 | .................................***X | 436.754 | | 415.954 | ................................X | 415.954 | | 390.354 | ..............................X | 390.354 | | 368.237 | ............................***...**.*X | 414.300 | | 414.180 | ................................X | 414.180 | | 406.880 | ...............................*..**...*.......**..*X | 586.386 | | 575.940 | .............................................X | 575.940 | | 542.640 | ..........................................*X | 542.640 | | 517.040 | ........................................*****..X | 545.000 | | 540.795 | ..........................................**X | 548.916 | | 513.516 | ........................................*X | 513.516 | | 415.183 | ................................**X | 415.183 | | 397.483 | ..............................X | 397.483 | | 393.700 | ..............................***.***..*..**..*.*X | 497.122 | | 492.422 | ......................................*X | 492.422 | | 491.125 | ......................................**..*X | 512.551 | | 490.926 | ......................................*X | 490.926 | | 486.291 | .....................................X | 486.291 | | 473.591 | ....................................**..X | 496.756 | | 478.126 | .....................................*X | 478.126 | | 445.626 | ..................................*X | 445.626 | | 413.916 | ................................X | 413.916 | | 396.780 | ..............................*.*X | 409.580 | | 402.118 | ...............................*X | 402.118 | | 357.018 | ...........................X | 357.018 | | 349.418 | ..........................**.*...*X | 391.051 | | 390.975 | ..............................****..****..***X | 448.877 | | 447.277 | ..................................*X | 447.277 | | 447.200 | ..................................**...***.X | 493.170 | | 440.520 | ..................................X | 440.520 | | 416.573 | ................................*X | 416.573 | | 381.573 | .............................**..X | 406.764 | | 394.710 | ..............................****...***.*.*..**..**X | 505.476 | | 478.276 | .....................................X | 478.276 | | 459.576 | ...................................***..***..*..*.**.*X | 561.130 | | 542.946 | ..........................................*X | 542.946 | | 500.739 | .......................................*X | 500.739 | | 453.268 | ...................................X | 453.268 | | 388.190 | ..............................*X | 388.190 | | 363.147 | ............................*X | 363.147 | | 335.547 | .........................X | 335.547 | | 269.769 | ....................*X | 269.769 | | 252.044 | ...................**......***..*..X | 382.026 | | 374.442 | ............................*X | 374.442 | | 356.542 | ...........................*X | 356.542 | | 349.358 | ..........................*..*.....**...**......X | 544.700 | | 529.500 | .........................................***..**..*X | 585.597 | | 582.597 | .............................................X | 582.597 | | 545.142 | ..........................................X | 545.142 | | 524.202 | ........................................*X | 524.202 | | 503.202 | .......................................X | 503.202 | | 479.584 | .....................................*X | 479.584 | | 449.654 | ..................................**.*.....**...*X | 560.013 | | 417.666 | ................................X | 417.666 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 23:35:45,052 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *.......**..........*...................*X | 12.000 | | 9.600 | ............................X | 9.600 | | 6.400 | ..................X | 6.400 | | 0.000 | *********************..*.......***.........................**X | 11.201 | | 8.801 | .........................X | 8.801 | | 5.600 | ................X | 5.600 | | 0.000 | ************************************.........**....**........X | 7.201 | | 6.401 | ..................*X | 6.401 | | 6.101 | .................*X | 6.101 | | 0.900 | .**X | 0.900 | | 0.000 | **********.*..**.*X | 1.500 | | 1.100 | ..X | 1.100 | | 0.300 | X | 0.300 | | 0.000 | **..**.....*..*X | 2.998 | | 2.198 | .....X | 2.198 | | 0.600 | ****........*X | 3.000 | | 2.400 | ......*****..*X | 3.000 | | 2.600 | ......X | 2.600 | | 1.800 | ....*X | 1.800 | | 1.500 | ...X | 1.500 | | 1.100 | ..X | 1.100 | | 0.300 | **X | 0.300 | | 0.000 | *.*..**.***...**..**...*X | 4.000 | | 2.800 | .......*X | 2.800 | | 1.600 | ...X | 1.600 | | 0.000 | ****......*............**....X | 7.504 | | 5.503 | ...............X | 5.503 | | 1.500 | ...*X | 1.500 | | 0.300 | **...*.....***X | 2.701 | | 1.901 | ....X | 1.901 | | 0.300 | X | 0.300 | | 0.000 | *****.**...*.....*X | 3.000 | | 2.700 | .......*X | 2.700 | | 2.300 | ......X | 2.300 | | 1.500 | ...X | 1.500 | | 1.200 | ..*X | 1.200 | | 0.800 | .X | 0.800 | | 0.000 | ********.*..**.*X | 1.500 | | 1.100 | ..X | 1.100 | | 0.300 | *X | 0.300 | | 0.000 | ***********..*..X | 1.500 | | 1.200 | ..*X | 1.200 | | 0.800 | .X | 0.800 | | 0.000 | *******************************..*.....**.....**......*.............*X | 10.199 | | 9.899 | .............................*X | 9.899 | | 4.701 | .............**X | 4.701 | | 3.501 | .........*X | 3.501 | | 1.900 | ....**.***...X | 3.000 | | 2.700 | .......**X | 2.700 | | 1.900 | ....X | 1.900 | | 0.300 | X | 0.300 | | 0.000 | ****************...*.....***X | 2.700 | | 2.699 | .......**.....**....*.......*X | 8.102 | | 6.002 | .................*X | 6.002 | | 4.000 | ...........X | 4.000 | | 0.000 | ***********...**...............**X | 6.001 | | 5.401 | ...............X | 5.401 | | 5.399 | ...............X | 5.399 | | 5.398 | ...............***X | 5.398 | | 3.800 | ..........X | 3.800 | | 0.600 | X | 0.600 | | 0.000 | **.**..*.*X | 1.500 | | 1.100 | ..X | 1.100 | | 0.300 | ****....**X | 1.500 | | 1.500 | ...*...*.................**......**....X | 11.201 | | 7.800 | ......................X | 7.800 | | 6.601 | ...................***X | 6.601 | | 6.201 | .................X | 6.201 | | 5.600 | ................**......*..................................**................*X | 24.089 | | 13.202 | .......................................**X | 13.202 | | 8.702 | .........................X | 8.702 | | 7.501 | .....................*.....**.*X | 9.202 | | 6.401 | ..................X | 6.401 | | 1.499 | ...X | 1.499 | | 0.000 | ****..*.....**......*...........*X | 7.900 | | 7.600 | ......................*X | 7.600 | | 4.800 | .............**X | 4.800 | | 3.600 | .........X | 3.600 | | 1.999 | .....X | 1.999 | | 0.000 | ******....**...........**......................*......**...........*..................................**............*X | 32.807 | | 31.607 | ...............................................................................................X | 31.607 | | 21.795 | .................................................................X | 21.795 | | 14.295 | ..........................................*X | 14.295 | | 13.092 | ......................................**....*..................**X | 20.109 | | 20.103 | ............................................................*X | 20.103 | | 15.903 | ...............................................*X | 15.903 | | 1.500 | ...*X | 1.500 | | 0.000 | **********************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-02 23:35:45,357 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 47.550052687 | 3693387776 | 2506285056 | 0 / 3909087232 | 126650 / 254442 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 38.3350730689 | 3693387776 | 2618609664 | 0 / 3909087232 | 126000 / 257600 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 49.9222395023 | 6078930944 | 4743712768 | 0 / 3758092288 | 126750 / 249463 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 46.4701318852 | 4201836544 | 3078033408 | 0 / 3758092288 | 126334 / 249912 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-02 23:35:58,900 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 23:37:41,230 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155530400 | 245821058 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155377360 | 283092514 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 433287904 | 449953511 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-02 23:37:41,232 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-02 23:37:51,141 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_in_out 2021-05-02 23:37:51,141 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_in_out 2021-05-02 23:37:58,825 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 23:37:59,173 | test | INFO | pool-6-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 22.0218440437 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 23.3019587891 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 14.8401826484 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 0.727546412444 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 16.649694501 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-02 23:37:59,227 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:38:04,494 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:38:09,092 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 23:38:10,154 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 1.13 2021-05-02 23:38:15,339 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 4.99 2021-05-02 23:38:20,950 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 10.1 2021-05-02 23:38:26,098 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 16.99 2021-05-02 23:38:31,319 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 23.71 2021-05-02 23:38:36,477 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-02 23:38:41,572 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 23:38:46,657 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 23:38:51,750 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-02 23:38:56,957 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 37.76 2021-05-02 23:39:02,066 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 43.62 2021-05-02 23:39:07,260 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 49.65 2021-05-02 23:39:12,392 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 54.91 2021-05-02 23:39:17,505 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 61.22 2021-05-02 23:39:17,967 | test | INFO | pool-6-thread-18 | [table_view:display:72] Keys failed in bucket2:e1ClP-5-903000:ce1ej9dwsY0xzf-6-43000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:17,967 | test | INFO | pool-6-thread-17 | [table_view:display:72] Keys failed in bucket2:e1ClP-5-903000:eDOlO6FnRJfpR-6-42000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:17,969 | test | INFO | pool-6-thread-18 | [table_view:display:72] Keys failed in bucket2:e1ClP-5-903000:ce1ej9dwsY0xzf-6-43000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2841 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2804 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2994 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:17,970 | test | INFO | pool-6-thread-17 | [table_view:display:72] Keys failed in bucket2:e1ClP-5-903000:eDOlO6FnRJfpR-6-42000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2841 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2804 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2994 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:22,618 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 23:39:27,713 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 23:39:32,826 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-02 23:39:38,262 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 69.69 2021-05-02 23:39:43,426 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 74.65 2021-05-02 23:39:45,667 | test | INFO | pool-6-thread-12 | [table_view:display:72] Keys failed in default:scope1:0qVsjmSjF52MBMeXv9vD-49-915000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:46,676 | test | INFO | pool-6-thread-14 | [table_view:display:72] Keys failed in default:scope1:2IXY4-49-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2592 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:46,677 | test | INFO | pool-6-thread-14 | [table_view:display:72] Keys failed in default:scope1:2IXY4-49-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2592 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:47,822 | test | INFO | pool-6-thread-11 | [table_view:display:72] Keys failed in default:scope1:U5fe8gJHmlM1j-49-914000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2727 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:47,825 | test | INFO | pool-6-thread-11 | [table_view:display:72] Keys failed in default:scope1:U5fe8gJHmlM1j-49-914000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2727 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:48,586 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.12 2021-05-02 23:39:50,687 | test | INFO | pool-6-thread-6 | [table_view:display:72] Keys failed in default:scope1:2IXY4-49-915000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:51,569 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:DF8_qtBNyO-8-133000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1657 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:51,569 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:DF8_qtBNyO-8-133000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1657 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:53,711 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 84.23 2021-05-02 23:39:57,361 | test | INFO | pool-6-thread-28 | [table_view:display:72] Keys failed in default:scope1:CspZwIDRPBYaxqaO-49-915000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-81 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,364 | test | INFO | pool-6-thread-25 | [table_view:display:72] Keys failed in default:scope1:cfqvcKjHInoRnZS-49-914000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-81 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,424 | test | INFO | pool-6-thread-30 | [table_view:display:72] Keys failed in default:scope1:4x6-49-916000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,426 | test | INFO | pool-6-thread-9 | [table_view:display:72] Keys failed in default:scope1:CspZwIDRPBYaxqaO-49-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,426 | test | INFO | pool-6-thread-23 | [table_view:display:72] Keys failed in default:scope1:cfqvcKjHInoRnZS-49-914000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,427 | test | INFO | pool-6-thread-30 | [table_view:display:72] Keys failed in default:scope1:4x6-49-916000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2690 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,427 | test | INFO | pool-6-thread-9 | [table_view:display:72] Keys failed in default:scope1:CspZwIDRPBYaxqaO-49-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2690 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,427 | test | INFO | pool-6-thread-23 | [table_view:display:72] Keys failed in default:scope1:cfqvcKjHInoRnZS-49-914000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2690 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:57,960 | test | INFO | pool-6-thread-11 | [table_view:display:72] Keys failed in default:scope1:cfqvcKjHInoRnZS-49-914000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-599 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:58,240 | test | INFO | pool-6-thread-14 | [table_view:display:72] Keys failed in default:scope1:2IXY4-49-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:58,242 | test | INFO | pool-6-thread-14 | [table_view:display:72] Keys failed in default:scope1:2IXY4-49-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2830 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:39:58,835 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 89.34 2021-05-02 23:39:59,812 | test | INFO | pool-6-thread-10 | [table_view:display:72] Keys failed in default:scope1:4x6-49-916000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-880 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:00,973 | test | INFO | pool-6-thread-8 | [table_view:display:72] Keys failed in default:scope1:cJW52JR-49-914000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-651 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:01,802 | test | INFO | pool-6-thread-22 | [table_view:display:72] Keys failed in default:scope1:cJW52JR-49-914000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:03,967 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 94.26 2021-05-02 23:40:05,605 | test | INFO | pool-6-thread-29 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:6-8-134000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1913 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1883 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:05,605 | test | INFO | pool-6-thread-29 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:6-8-134000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1913 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1883 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:06,605 | test | INFO | pool-6-thread-14 | [table_view:display:72] Keys failed in default:scope1:sTvkf-49-916000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-574 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:09,078 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 99.11 2021-05-02 23:40:11,098 | test | INFO | pool-6-thread-23 | [table_view:display:72] Keys failed in default:scope1:sTvkf-49-916000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:11,099 | test | INFO | pool-6-thread-23 | [table_view:display:72] Keys failed in default:scope1:sTvkf-49-916000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2504 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:11,815 | test | INFO | pool-6-thread-2 | [table_view:display:72] Keys failed in default:scope1:eM-49-915000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-30 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-75 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:11,891 | test | INFO | pool-6-thread-20 | [table_view:display:72] Keys failed in default:scope1:eM-49-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:11,892 | test | INFO | pool-6-thread-20 | [table_view:display:72] Keys failed in default:scope1:eM-49-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2621 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2664 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:12,088 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:DF8_qtBNyO-8-133000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:12,088 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:DF8_qtBNyO-8-133000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2049 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:12,306 | test | INFO | pool-6-thread-29 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:6-8-134000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:12,308 | test | INFO | pool-6-thread-29 | [table_view:display:72] Keys failed in default:xVVeJJ-7-992000:6-8-134000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2116 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2086 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:12,489 | test | INFO | pool-6-thread-6 | [table_view:display:72] Keys failed in default:scope1:eM-49-915000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-694 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:13,816 | test | INFO | pool-6-thread-14 | [table_view:display:72] Keys failed in default:scope1:sTvkf-49-916000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-757 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:40:14,256 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-02 23:40:19,413 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-02 23:40:29,762 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172300 type=client open_count=1 channel=[id: 0xf41649c8, 0.0.0.0/0.0.0.0:34531] timeout=300.0>) 2021-05-02 23:40:29,815 | test | ERROR | pool-6-thread-7 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17233b type=client open_count=1 channel=[id: 0x06c88024, 0.0.0.0/0.0.0.0:43507] timeout=300.0>) 2021-05-02 23:40:31,834 | test | ERROR | pool-6-thread-7 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 23:40:34,020 | test | INFO | pool-6-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 154.848999977 sec 2021-05-02 23:42:58,443 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-02 23:42:58,444 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 4 docs from 0 to 5000 2021-05-02 23:42:58,444 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 3 docs from 0 to 5000 2021-05-02 23:42:58,446 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 3 docs from 0 to 5000 2021-05-02 23:42:58,459 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:42:58,460 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:42:58,461 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:42:58,461 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-02 23:42:58,463 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:42:58,464 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:42:58,464 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:42:58,466 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:42:58,466 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:42:58,467 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:42:58,467 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:42:58,467 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:42:58,469 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-02 23:42:58,470 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-02 23:42:58,470 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-02 23:42:58,470 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-02 23:42:58,470 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-02 23:42:58,471 | test | ERROR | pool-6-thread-3 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-02 23:50:19,923 | test | INFO | pool-6-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****.*X | 5.001 | | 4.399 | X | 4.399 | | 3.399 | X | 3.399 | | 2.199 | **.*.*.*X | 14.504 | | 10.000 | .X | 10.000 | | 3.000 | *X | 3.000 | | 0.000 | ***************************************..**..................*.........................................*.............................**..........X | 462.319 | | 456.600 | .................................................................................................*X | 456.600 | | 367.700 | ..............................................................................*X | 367.700 | | 227.700 | ................................................**X | 227.700 | | 226.600 | ................................................X | 226.600 | | 225.100 | ...............................................X | 225.100 | | 185.700 | .......................................*X | 185.700 | | 166.200 | ..................................*X | 166.200 | | 155.200 | ................................X | 155.200 | | 137.100 | ............................X | 137.100 | | 122.000 | .........................*X | 122.000 | | 114.200 | .......................*.**......**..*...*X | 169.900 | | 154.000 | ................................**..X | 162.400 | | 146.293 | ..............................**X | 146.393 | | 129.693 | ...........................*....*.**....*X | 175.209 | | 161.100 | .................................*...**..*.............*X | 240.500 | | 225.800 | ...............................................*X | 225.800 | | 217.976 | ..............................................***.........X | 262.934 | | 257.910 | ......................................................*.......*X | 290.276 | | 270.351 | .........................................................**.X | 275.401 | | 259.759 | .......................................................X | 259.759 | | 234.826 | .................................................*X | 234.826 | | 180.000 | .....................................X | 180.000 | | 65.400 | .............*X | 65.400 | | 25.300 | ....*X | 25.300 | | 15.200 | ..X | 15.200 | | 0.100 | **X | 0.200 | | 0.100 | **X | 0.100 | | 0.000 | ******************************X | 0.300 | | 0.100 | *X | 0.100 | | 0.000 | ********************X | 0.200 | | 0.100 | X | 0.100 | | 0.000 | ****************X | 0.400 | | 0.200 | X | 0.200 | | 0.000 | *******X | 1.000 | | 0.800 | X | 0.800 | | 0.300 | *X | 0.300 | | 0.100 | *X | 0.100 | | 0.000 | **************************X | 0.500 | | 0.400 | X | 0.400 | | 0.200 | *X | 0.200 | | 0.000 | ***************X | 0.400 | | 0.200 | ****X | 0.300 | | 0.100 | *X | 0.100 | | 0.000 | ******************X | 0.200 | | 0.100 | *X | 0.100 | | 0.000 | ******************************X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 23:50:21,243 | test | INFO | pool-6-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***********....*.........*....................X | 391.934 | | 391.934 | ................................*..............*...........................X | 879.044 | | 879.044 | .........................................................................X | 879.044 | | 763.044 | ...............................................................X | 763.044 | | 666.803 | .......................................................X | 666.803 | | 546.360 | .............................................X | 546.360 | | 540.360 | ............................................**..........**.....X | 720.246 | | 694.427 | .........................................................X | 694.427 | | 532.627 | ............................................*X | 532.627 | | 461.427 | ......................................*X | 461.427 | | 413.800 | ..................................**......*.....*.............X | 708.100 | | 645.500 | .....................................................*X | 645.500 | | 599.537 | .................................................X | 599.537 | | 486.028 | ........................................*...*....*............*X | 712.829 | | 688.176 | .........................................................X | 688.176 | | 655.692 | ......................................................X | 655.692 | | 497.946 | .........................................*......**.......*...........................**.*X | 990.200 | | 971.000 | .................................................................................X | 971.000 | | 862.963 | ........................................................................X | 862.963 | | 808.763 | ...................................................................*X | 808.763 | | 654.963 | ......................................................X | 654.963 | | 455.925 | .....................................X | 455.925 | | 344.025 | ............................*X | 344.025 | | 320.025 | ..........................X | 320.025 | | 210.200 | ................*X | 210.200 | | 181.200 | ..............*X | 181.200 | | 136.700 | ..........***.*..*.....**..*X | 249.500 | | 237.000 | ...................X | 237.000 | | 156.600 | ............X | 156.600 | | 155.700 | ............**......**................*.*X | 436.900 | | 412.600 | .................................X | 412.600 | | 308.500 | .........................*X | 308.500 | | 297.200 | ........................*X | 297.200 | | 246.186 | ...................**X | 246.186 | | 237.586 | ...................*..X | 261.186 | | 200.809 | ...............**..**..........**..*X | 356.600 | | 344.600 | ............................*X | 344.600 | | 304.000 | ........................X | 304.000 | | 246.400 | ...................*X | 246.400 | | 241.900 | ...................***....**...*X | 327.176 | | 322.753 | ..........................*X | 322.753 | | 311.314 | .........................*X | 311.314 | | 290.465 | .......................*......**........*.................**..**.....***......X | 808.170 | | 784.670 | .................................................................X | 784.670 | | 704.370 | ..........................................................*X | 704.370 | | 606.683 | ..................................................*X | 606.683 | | 504.622 | .........................................*X | 504.622 | | 494.138 | ........................................*X | 494.138 | | 456.547 | .....................................**...*........X | 579.747 | | 579.747 | ................................................*....*............*X | 769.839 | | 676.439 | ........................................................*X | 676.439 | | 478.469 | .......................................**.....**..*........*X | 660.182 | | 585.813 | ................................................*X | 585.813 | | 476.363 | .......................................X | 476.363 | | 282.231 | ......................X | 282.231 | | 257.852 | ....................**...*..***..*...*............**..*X | 540.663 | | 507.025 | .........................................X | 507.025 | | 345.894 | ............................*X | 345.894 | | 283.294 | ......................**......**......*X | 420.736 | | 411.500 | .................................X | 411.500 | | 372.600 | ..............................*X | 372.600 | | 305.169 | ........................*X | 305.169 | | 285.669 | .......................X | 285.669 | | 235.569 | ..................**..*...*.......*X | 366.444 | | 366.396 | ..............................*X | 366.396 | | 362.996 | .............................*..**.*X | 400.945 | | 316.630 | .........................**X | 316.630 | | 297.430 | ........................*.*...*X | 348.801 | | 341.249 | ...........................X | 341.249 | | 341.249 | ...........................*.............***..........*..*............**.*.........X | 891.500 | | 856.819 | .......................................................................*X | 856.819 | | 838.619 | .....................................................................*X | 838.619 | | 771.819 | ................................................................*....*.......*X | 904.797 | | 892.922 | ..........................................................................X | 892.922 | | 836.335 | .....................................................................*X | 836.335 | | 746.553 | ..............................................................**......*.............*....*X | 1023.806 | | 1004.700 | ....................................................................................***.....**..........*X | 1181.400 | | 1175.200 | ..................................................................................................*X | 1175.200 | | 1137.300 | ...............................................................................................X | 1137.300 | | 1077.900 | ..........................................................................................*X | 1077.900 | | 1038.300 | ......................................................................................*X | 1038.300 | | 940.060 | ..............................................................................**....*X | 981.360 | | 974.860 | .................................................................................*.........*X | 1079.752 | | 980.752 | ..................................................................................*X | 980.752 | | 931.452 | .............................................................................X | 931.452 | | 739.863 | .............................................................X | 739.863 | | 677.563 | ........................................................*X | 677.563 | | 589.463 | ................................................X | 589.463 | | 468.722 | ......................................*X | 468.722 | | 443.922 | ....................................*X | 443.922 | | 423.822 | ..................................*....**....*X | 517.400 | | 474.729 | .......................................*X | 474.729 | | 442.929 | ....................................**...X | 478.829 | | 467.262 | ......................................*X | 467.262 | | 449.362 | .....................................*****....*..**X | 530.906 | | 460.976 | ......................................*X | 460.976 | | 442.856 | ....................................*....*.....*X | 553.092 | | 506.454 | .........................................X | 506.454 | | 383.954 | ...............................*X | 383.954 | | 363.408 | .............................**.........**X | 472.508 | | 445.371 | ....................................X | 445.371 | | 436.671 | ...................................*...**....*X | 514.000 | | 478.900 | .......................................X | 478.900 | | 441.600 | ....................................X | 441.600 | | 397.500 | ................................**....***..*X | 466.000 | | 434.600 | ...................................*....*X | 475.800 | | 452.545 | .....................................*X | 452.545 | +----------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-02 23:50:22,621 | test | INFO | pool-6-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***********.............*..........................**..........................**...................................*X | 175.008 | | 159.608 | ..........................................................................................*X | 159.608 | | 133.167 | ...........................................................................**.....*X | 143.227 | | 135.427 | ............................................................................*X | 135.427 | | 125.627 | ......................................................................*......X | 135.608 | | 135.208 | ............................................................................*.....*X | 144.808 | | 144.800 | .................................................................................*..............***X | 168.800 | | 161.200 | ...........................................................................................*X | 161.200 | | 151.200 | .....................................................................................X | 151.200 | | 121.195 | ....................................................................X | 121.195 | | 48.793 | ..........................X | 48.793 | | 18.793 | .........X | 18.793 | | 8.798 | ....X | 8.798 | | 0.000 | ********************.......*.............**.........*X | 52.200 | | 49.400 | ...........................*...*X | 55.200 | | 53.100 | .............................*X | 53.100 | | 51.800 | ............................**X | 51.800 | | 46.800 | .........................X | 46.800 | | 41.800 | ......................*....****...X | 52.800 | | 46.600 | .........................***X | 46.600 | | 41.600 | ......................*.*...****.***...*...X | 60.003 | | 59.903 | .................................***X | 59.903 | | 51.200 | ............................X | 51.200 | | 46.400 | .........................***.....*X | 55.800 | | 55.700 | ..............................*X | 55.700 | | 12.999 | ......**X | 12.999 | | 2.999 | *X | 2.999 | | 0.000 | ************X | 1.200 | | 0.700 | X | 0.700 | | 0.200 | **X | 0.200 | | 0.100 | *X | 0.100 | | 0.000 | *******X | 0.500 | | 0.300 | ***X | 0.300 | | 0.100 | X | 0.100 | | 0.000 | ********......**...............**............*..*..*X | 66.386 | | 61.111 | .................................**...*..**.....**............*X | 98.125 | | 88.325 | .................................................*X | 88.325 | | 83.017 | ..............................................X | 83.017 | | 78.501 | ...........................................**......**X | 88.317 | | 85.908 | ................................................*X | 85.908 | | 80.908 | .............................................*X | 80.908 | | 80.900 | .............................................X | 80.900 | | 80.700 | .............................................*..*X | 85.688 | | 85.588 | ...............................................X | 85.588 | | 83.788 | ..............................................**X | 83.788 | | 73.790 | .........................................*...*.........**.........*X | 111.137 | | 110.537 | ..............................................................X | 110.537 | | 103.323 | ..........................................................X | 103.323 | | 87.686 | .................................................*X | 87.686 | | 79.086 | ............................................X | 79.086 | | 66.908 | .....................................**...**..*...**...X | 85.883 | | 81.490 | .............................................X | 81.490 | | 65.086 | ....................................*X | 65.086 | | 40.704 | ......................X | 40.704 | | 40.704 | ......................X | 40.704 | | 26.300 | ..............X | 26.300 | | 4.000 | .**X | 4.600 | | 3.900 | .X | 3.900 | | 1.900 | *X | 1.900 | | 1.000 | *X | 1.000 | | 0.900 | **X | 1.300 | | 1.000 | X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ******X | 1.600 | | 1.500 | **.****.***.***.*X | 7.100 | | 6.100 | ..X | 6.100 | | 5.400 | ..X | 5.400 | | 3.200 | *X | 3.200 | | 3.100 | ****X | 3.100 | | 2.700 | **.**......**...........*..........................**....*X | 85.804 | | 67.000 | .....................................***.....*......*.......*X | 98.485 | | 97.785 | ......................................................**...*..**.*X | 106.815 | | 97.615 | ......................................................X | 97.615 | | 95.791 | .....................................................X | 95.791 | | 92.791 | ....................................................*X | 92.791 | | 92.085 | ...................................................X | 92.085 | | 88.904 | .................................................**.*X | 89.394 | | 89.007 | .................................................X | 89.007 | | 80.497 | ............................................X | 80.497 | | 72.812 | ........................................*X | 72.812 | | 69.106 | ......................................*......**......*........X | 104.979 | | 103.479 | ..........................................................*X | 103.479 | | 93.313 | ....................................................*X | 93.313 | | 78.913 | ............................................X | 78.913 | | 72.013 | ........................................*...**.X | 78.900 | | 72.900 | ........................................*X | 72.900 | | 66.600 | .....................................*X | 66.600 | | 65.700 | ....................................*...***X | 71.600 | | 60.800 | .................................X | 60.800 | | 29.100 | ...............X | 29.100 | | 14.100 | .......*X | 14.100 | | 7.400 | ...X | 7.400 | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 23:50:22,970 | test | WARNING | Thread-11301 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:U5fe8gJHmlM1j-49-914000 create failed keys from task: [u'test_collections-2727'] 2021-05-02 23:50:22,970 | test | WARNING | Thread-11303 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:2IXY4-49-915000 delete failed keys from task: [u'test_collections-264'] 2021-05-02 23:50:22,970 | test | WARNING | Thread-11305 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:4x6-49-916000 update failed keys from task: [u'test_collections-880'] 2021-05-02 23:50:22,970 | test | WARNING | Thread-11306 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:cfqvcKjHInoRnZS-49-914000 update failed keys from task: [u'test_collections-599'] 2021-05-02 23:50:22,971 | test | WARNING | Thread-11308 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:cJW52JR-49-914000 update failed keys from task: [u'test_collections-651'] 2021-05-02 23:50:22,970 | test | WARNING | Thread-11302 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:0qVsjmSjF52MBMeXv9vD-49-915000 delete failed keys from task: [u'test_collections-2', u'test_collections-126'] 2021-05-02 23:50:23,015 | test | WARNING | Thread-11308 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:cJW52JR-49-914000 delete failed keys from task: [u'test_collections-7'] 2021-05-02 23:50:23,015 | test | WARNING | Thread-11303 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:2IXY4-49-915000 create failed keys from task: [u'test_collections-2830', u'test_collections-2592'] 2021-05-02 23:50:23,016 | test | WARNING | Thread-11306 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:cfqvcKjHInoRnZS-49-914000 delete failed keys from task: [u'test_collections-81'] 2021-05-02 23:50:23,017 | test | WARNING | Thread-11507 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:e1ClP-5-903000:eDOlO6FnRJfpR-6-42000 create failed keys from task: [u'test_collections-2841', u'test_collections-2804', u'test_collections-2994'] 2021-05-02 23:50:23,019 | test | WARNING | Thread-11306 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:cfqvcKjHInoRnZS-49-914000 create failed keys from task: [u'test_collections-2690'] 2021-05-02 23:50:22,970 | test | WARNING | Thread-11307 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:CspZwIDRPBYaxqaO-49-915000 delete failed keys from task: [u'test_collections-81'] 2021-05-02 23:50:23,015 | test | WARNING | Thread-11305 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:4x6-49-916000 create failed keys from task: [u'test_collections-2690'] 2021-05-02 23:50:23,019 | test | WARNING | Thread-11508 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:e1ClP-5-903000:ce1ej9dwsY0xzf-6-43000 create failed keys from task: [u'test_collections-2841', u'test_collections-2804', u'test_collections-2994'] 2021-05-02 23:50:23,025 | test | WARNING | Thread-11307 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:CspZwIDRPBYaxqaO-49-915000 create failed keys from task: [u'test_collections-2690'] 2021-05-02 23:50:23,028 | test | WARNING | Thread-11310 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:sTvkf-49-916000 update failed keys from task: [u'test_collections-607', u'test_collections-757', u'test_collections-574'] 2021-05-02 23:50:23,028 | test | WARNING | Thread-11311 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:eM-49-915000 update failed keys from task: [u'test_collections-694'] 2021-05-02 23:50:23,032 | test | WARNING | Thread-11311 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:eM-49-915000 delete failed keys from task: [u'test_collections-30', u'test_collections-75'] 2021-05-02 23:50:23,036 | test | WARNING | Thread-11310 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:sTvkf-49-916000 create failed keys from task: [u'test_collections-2504'] 2021-05-02 23:50:23,039 | test | WARNING | Thread-11311 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:eM-49-915000 create failed keys from task: [u'test_collections-2621', u'test_collections-2664'] 2021-05-02 23:50:23,055 | test | WARNING | Thread-11358 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:xVVeJJ-7-992000:6-8-134000 create failed keys from task: [u'test_collections-2116', u'test_collections-1913', u'test_collections-2086', u'test_collections-1883'] 2021-05-02 23:50:23,058 | test | WARNING | Thread-11357 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:xVVeJJ-7-992000:DF8_qtBNyO-8-133000 create failed keys from task: [u'test_collections-2049', u'test_collections-1657'] 2021-05-02 23:50:34,487 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 23:51:54,068 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-02 23:51:54,068 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-02 23:52:11,654 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-02 23:52:11,657 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-02 23:52:29,921 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-02 23:52:29,923 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-02 23:53:14,743 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-02 23:53:31,844 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1530001163 seconds to complete 2021-05-02 23:53:31,864 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2339.713s OK During the test, Remote Connections: 162, Disconnections: 162 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_4 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_5', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 5, 'bulk_api_crud': 'True'} test_rebalance_cycles (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-02 23:53:41,749 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #5 test_rebalance_cycles ========= 2021-05-02 23:53:42,730 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #5 test_rebalance_cycles ========= 2021-05-02 23:53:43,263 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd0 type=client open_count=1 channel=[id: 0x8781e282, 0.0.0.0/0.0.0.0:33557] timeout=300.0>) 2021-05-02 23:53:48,756 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd1 type=client open_count=1 channel=[id: 0xd970150a, 0.0.0.0/0.0.0.0:35619] timeout=300.0>) 2021-05-02 23:53:50,779 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd2 type=client open_count=1 channel=[id: 0xca57c707, 0.0.0.0/0.0.0.0:40647] timeout=300.0>) 2021-05-02 23:53:52,802 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd3 type=client open_count=1 channel=[id: 0x8397716c, 0.0.0.0/0.0.0.0:53829] timeout=300.0>) 2021-05-02 23:53:54,825 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd4 type=client open_count=1 channel=[id: 0xc9cd1054, 0.0.0.0/0.0.0.0:57665] timeout=300.0>) 2021-05-02 23:53:56,846 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd5 type=client open_count=1 channel=[id: 0x6a248b09, 0.0.0.0/0.0.0.0:48507] timeout=300.0>) 2021-05-02 23:53:58,868 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd6 type=client open_count=1 channel=[id: 0x162c9693, 0.0.0.0/0.0.0.0:36485] timeout=300.0>) 2021-05-02 23:54:00,890 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd7 type=client open_count=1 channel=[id: 0x1cfd3478, 0.0.0.0/0.0.0.0:38535] timeout=300.0>) 2021-05-02 23:54:02,913 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x172fd8 type=client open_count=1 channel=[id: 0x21512b94, 0.0.0.0/0.0.0.0:52435] timeout=300.0>) 2021-05-02 23:54:04,938 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fd9 type=client open_count=1 channel=[id: 0x6572c5e3, 0.0.0.0/0.0.0.0:59860] timeout=300.0>) 2021-05-02 23:54:12,237 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fda type=client open_count=1 channel=[id: 0xcc1d34e8, 0.0.0.0/0.0.0.0:35423] timeout=300.0>) 2021-05-02 23:54:14,265 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fdb type=client open_count=1 channel=[id: 0x4487209d, 0.0.0.0/0.0.0.0:55571] timeout=300.0>) 2021-05-02 23:54:16,286 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fdc type=client open_count=1 channel=[id: 0x133fc76d, 0.0.0.0/0.0.0.0:42370] timeout=300.0>) 2021-05-02 23:54:18,309 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fdd type=client open_count=1 channel=[id: 0x3de5df0e, 0.0.0.0/0.0.0.0:37456] timeout=300.0>) 2021-05-02 23:54:20,332 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fde type=client open_count=1 channel=[id: 0xbc227bbd, 0.0.0.0/0.0.0.0:44117] timeout=300.0>) 2021-05-02 23:54:22,357 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fdf type=client open_count=1 channel=[id: 0xfe763bf2, 0.0.0.0/0.0.0.0:44893] timeout=300.0>) 2021-05-02 23:54:24,378 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fe0 type=client open_count=1 channel=[id: 0xa05143b7, 0.0.0.0/0.0.0.0:53991] timeout=300.0>) 2021-05-02 23:54:26,401 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x172fe1 type=client open_count=1 channel=[id: 0xd856ab05, 0.0.0.0/0.0.0.0:60057] timeout=300.0>) 2021-05-02 23:54:28,424 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe2 type=client open_count=1 channel=[id: 0x7254d291, 0.0.0.0/0.0.0.0:50911] timeout=300.0>) 2021-05-02 23:54:35,746 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe3 type=client open_count=1 channel=[id: 0x85745ad1, 0.0.0.0/0.0.0.0:34386] timeout=300.0>) 2021-05-02 23:54:37,769 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe4 type=client open_count=1 channel=[id: 0xf6042cd1, 0.0.0.0/0.0.0.0:58895] timeout=300.0>) 2021-05-02 23:54:39,790 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe5 type=client open_count=1 channel=[id: 0x3cfd1d40, 0.0.0.0/0.0.0.0:59815] timeout=300.0>) 2021-05-02 23:54:41,816 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe6 type=client open_count=1 channel=[id: 0xdf64277e, 0.0.0.0/0.0.0.0:47315] timeout=300.0>) 2021-05-02 23:54:43,841 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe7 type=client open_count=1 channel=[id: 0x45ee8fad, 0.0.0.0/0.0.0.0:57679] timeout=300.0>) 2021-05-02 23:54:45,868 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe8 type=client open_count=1 channel=[id: 0xc3c48c2c, 0.0.0.0/0.0.0.0:45574] timeout=300.0>) 2021-05-02 23:54:47,890 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fe9 type=client open_count=1 channel=[id: 0xbd739323, 0.0.0.0/0.0.0.0:39933] timeout=300.0>) 2021-05-02 23:54:49,911 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172fea type=client open_count=1 channel=[id: 0x5a339d07, 0.0.0.0/0.0.0.0:52489] timeout=300.0>) 2021-05-02 23:54:51,940 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x172feb type=client open_count=1 channel=[id: 0x20249b43, 0.0.0.0/0.0.0.0:44935] timeout=300.0>) 2021-05-02 23:54:53,963 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172fec type=client open_count=1 channel=[id: 0x322a3c78, 0.0.0.0/0.0.0.0:50889] timeout=300.0>) 2021-05-02 23:55:02,107 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172fed type=client open_count=1 channel=[id: 0xdd750f1e, 0.0.0.0/0.0.0.0:60897] timeout=300.0>) 2021-05-02 23:55:04,132 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172fee type=client open_count=1 channel=[id: 0x2fa95c7f, 0.0.0.0/0.0.0.0:44340] timeout=300.0>) 2021-05-02 23:55:06,154 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172fef type=client open_count=1 channel=[id: 0x9a9f3b0b, 0.0.0.0/0.0.0.0:51816] timeout=300.0>) 2021-05-02 23:55:08,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172ff0 type=client open_count=1 channel=[id: 0x3281a6d4, 0.0.0.0/0.0.0.0:34609] timeout=300.0>) 2021-05-02 23:55:10,201 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172ff1 type=client open_count=1 channel=[id: 0xa7a9a36d, 0.0.0.0/0.0.0.0:42363] timeout=300.0>) 2021-05-02 23:55:12,224 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172ff2 type=client open_count=1 channel=[id: 0xf58de752, 0.0.0.0/0.0.0.0:42597] timeout=300.0>) 2021-05-02 23:55:14,259 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172ff3 type=client open_count=1 channel=[id: 0x68ddf729, 0.0.0.0/0.0.0.0:42504] timeout=300.0>) 2021-05-02 23:55:16,285 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x172ff4 type=client open_count=1 channel=[id: 0xe42d41a6, 0.0.0.0/0.0.0.0:57693] timeout=300.0>) 2021-05-02 23:55:18,309 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x172ff5 type=client open_count=1 channel=[id: 0x73821e3c, 0.0.0.0/0.0.0.0:32873] timeout=300.0>) 2021-05-02 23:55:23,224 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x172ff6 type=client open_count=1 channel=[id: 0xbce2ebb1, 0.0.0.0/0.0.0.0:47537] timeout=300.0>) 2021-05-02 23:55:25,249 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x172ff7 type=client open_count=1 channel=[id: 0x4ad8998e, 0.0.0.0/0.0.0.0:58941] timeout=300.0>) 2021-05-02 23:55:27,288 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x172ff8 type=client open_count=1 channel=[id: 0xdecdde8f, 0.0.0.0/0.0.0.0:46377] timeout=300.0>) 2021-05-02 23:55:29,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x172ff9 type=client open_count=1 channel=[id: 0x1d958658, 0.0.0.0/0.0.0.0:55168] timeout=300.0>) 2021-05-02 23:55:31,335 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x172ffa type=client open_count=1 channel=[id: 0x2a216c5b, 0.0.0.0/0.0.0.0:60177] timeout=300.0>) 2021-05-02 23:55:33,358 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-02 23:55:50,487 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-02 23:56:05,150 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-02 23:56:05,198 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #5 test_rebalance_cycles ========= 2021-05-02 23:56:05,242 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #5 test_rebalance_cycles ========= 2021-05-02 23:56:37,022 | test | INFO | pool-7-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.98260650366 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-02 23:56:37,051 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:56:42,117 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:56:52,201 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-02 23:56:52,236 | test | INFO | pool-7-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.2139999866 sec 2021-05-02 23:56:52,624 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-02 23:56:52,868 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-02 23:56:53,069 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | VG-52-682000 | couchbase | 2 | none | 0 | 0 | 10825498624 | 0 | 0 | +--------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-02 23:56:53,071 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-02 23:56:54,457 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 23:56:54,466 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 23:57:24,838 | test | INFO | pool-7-thread-9 | [table_view:display:72] Ops trend for bucket 'VG-52-682000' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 118.700 | ............................***.......................*................................................*X | 401.100 | | 382.400 | ..............................................................................................**.X | 387.400 | | 382.600 | ..............................................................................................X | 382.600 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-02 23:57:24,905 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 20.3137254902 | 3693387776 | 2975473664 | 0 / 3909087232 | 2001 / 4010 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 22.0476564546 | 3693387776 | 3048988672 | 0 / 3909087232 | 1599 / 3219 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 20.2651032373 | 6078930944 | 5208559616 | 0 / 3758092288 | 2151 / 4242 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 25.3517457009 | 4201836544 | 3559403520 | 0 / 3758092288 | 2149 / 4346 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-02 23:57:34,019 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-02 23:57:55,855 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | VG-52-682000 | couchbase | 2 | none | 0 | 10000 | 10825498624 | 103923744 | 178484619 | +--------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-02 23:57:55,858 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #5 test_rebalance_cycles ========= 2021-05-02 23:58:02,536 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 0 2021-05-02 23:58:02,683 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-02 23:58:02,690 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-02 23:58:02,753 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-02 23:58:04,793 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-02 23:58:06,844 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-02 23:58:08,881 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-02 23:58:10,523 | test | INFO | pool-7-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 9.64402928553 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 17.5447441391 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 10.2467270896 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 11.4379913771 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-02 23:58:10,539 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:58:11,003 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-02 23:58:15,638 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:58:20,710 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:58:25,783 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-02 23:58:30,848 | test | ERROR | pool-7-thread-17 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'072936ba1c3c193c7d8610c56757bedb', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=1ae5320e1248720433ca8b05e521ff96', u'status': u'notRunning'} - rebalance failed 2021-05-02 23:58:31,163 | test | INFO | pool-7-thread-17 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.98.196: 2021-05-02 23:58:31,164 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1620025105878L, u'shortText': u'message', u'serverTime': u'2021-05-02T23:58:25.878Z', u'text': u"Rebalance exited with reason {buckets_cleanup_failed,['ns_1@172.23.104.186']}.\nRebalance Operation Id = 0c7edaeac725974d21c90107c8baf059"} 2021-05-02 23:58:31,165 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.98.196', u'tstamp': 1620025105876L, u'shortText': u'message', u'serverTime': u'2021-05-02T23:58:25.876Z', u'text': u"Failed to cleanup old buckets on node 'ns_1@172.23.104.186': {badrpc,\n {'EXIT',\n timeout}}"} 2021-05-02 23:58:31,167 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1620025090455L, u'shortText': u'message', u'serverTime': u'2021-05-02T23:58:10.455Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.98.196','ns_1@172.23.98.195',\n 'ns_1@172.23.104.186','ns_1@172.23.120.206',\n 'ns_1@172.23.121.10'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 0c7edaeac725974d21c90107c8baf059"} 2021-05-02 23:58:31,167 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.120.206', u'tstamp': 1620025090376L, u'shortText': u'message', u'serverTime': u'2021-05-02T23:58:10.376Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'} 2021-05-02 23:58:31,168 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.120.206', u'tstamp': 1620025090289L, u'shortText': u'message', u'serverTime': u'2021-05-02T23:58:10.289Z', u'text': u'Node ns_1@172.23.120.206 joined cluster'} 2021-05-02 23:58:31,170 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.120.206', u'tstamp': 1620025090245L, u'shortText': u'web start ok', u'serverTime': u'2021-05-02T23:58:10.245Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.120.206\'. Version: "7.0.0-5085-enterprise".'} 2021-05-02 23:58:31,170 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.121.10', u'tstamp': 1620025083292L, u'shortText': u'node up', u'serverTime': u'2021-05-02T23:58:03.292Z', u'text': u"Node 'ns_1@172.23.121.10' saw that node 'ns_1@172.23.120.206' came up. Tags: []"} 2021-05-02 23:58:31,171 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.104.186', u'tstamp': 1620025083286L, u'shortText': u'node up', u'serverTime': u'2021-05-02T23:58:03.286Z', u'text': u"Node 'ns_1@172.23.104.186' saw that node 'ns_1@172.23.120.206' came up. Tags: []"} 2021-05-02 23:58:31,171 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.98.195', u'tstamp': 1620025083282L, u'shortText': u'node up', u'serverTime': u'2021-05-02T23:58:03.282Z', u'text': u"Node 'ns_1@172.23.98.195' saw that node 'ns_1@172.23.120.206' came up. Tags: []"} 2021-05-02 23:58:31,171 | test | ERROR | pool-7-thread-17 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_config', u'type': u'info', u'node': u'ns_1@172.23.98.196', u'tstamp': 1620025083092L, u'shortText': u'message', u'serverTime': u'2021-05-02T23:58:03.092Z', u'text': u'Conflicting configuration changes to field scramsha_fallback_salt:\n<<49,104,134,70,82,193,45,98,227,31,67,143>> and\n<<80,67,29,226,247,192,11,116,164,212,182,63>>, choosing the former.'} 2021-05-02 23:58:31,173 | test | ERROR | pool-7-thread-17 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'072936ba1c3c193c7d8610c56757bedb', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=1ae5320e1248720433ca8b05e521ff96', u'status': u'notRunning'} - rebalance failed FAIL 2021-05-02 23:58:32,546 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: VG-52-682000, Active Resident ratio(DGM): 100% 2021-05-02 23:58:32,548 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: VG-52-682000, Replica Resident ratio(DGM): 100% 2021-05-02 23:58:33,240 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in VG-52-682000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1104 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1184 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1020 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1141 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1065 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1144 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1101 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1025 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1181 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1060 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1051 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1054 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1175 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1011 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1014 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1135 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1091 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1094 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1169 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1048 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1007 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1129 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1008 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1163 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1042 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1087 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1088 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1166 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1002 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1123 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1047 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1082 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1158 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1039 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1117 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1118 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1073 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1076 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1197 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1033 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1198 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1112 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1079 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1157 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1036 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-02 23:58:57,855 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-02 23:58:58,000 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1085] Running cbcollect on node ns_1@172.23.104.186,ns_1@172.23.120.206,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 2021-05-02 23:59:08,051 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1088] ns_1@172.23.104.186,ns_1@172.23.120.206,ns_1@172.23.121.10,ns_1@172.23.98.195,ns_1@172.23.98.196 - cbcollect status: True 2021-05-02 23:59:08,052 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1093] Polling active_tasks to check cbcollect status 2021-05-03 00:03:29,223 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.104.186: Copying cbcollect ZIP file to Client 2021-05-03 00:03:31,319 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.120.206: Copying cbcollect ZIP file to Client 2021-05-03 00:03:32,903 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.121.10: Copying cbcollect ZIP file to Client 2021-05-03 00:03:35,256 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.98.195: Copying cbcollect ZIP file to Client 2021-05-03 00:03:37,493 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1120] ns_1@172.23.98.196: Copying cbcollect ZIP file to Client 2021-05-03 00:03:40,668 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:491] Skipping get_trace !! 2021-05-03 00:03:53,865 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2100000381 seconds to complete 2021-05-03 00:03:53,884 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete During the test, Remote Connections: 67, Disconnections: 67 SDK Connections: 1, Disconnections: 1 ====================================================================== FAIL: test_rebalance_cycles (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 950, in test_rebalance_cycles self.wait_for_rebalance_to_complete(operation) File "pytests/bucket_collections/collections_rebalance.py", line 792, in wait_for_rebalance_to_complete self.assertTrue(task.result, "Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 622.141s FAILED (failures=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 1 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_5 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,graceful=False' Test Input params: {'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'graceful': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_6', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 6, 'bulk_api_crud': 'True'} test_Orchestrator_Node_failover (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 00:04:04,052 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 00:04:05,032 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 00:04:05,651 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173370 type=client open_count=1 channel=[id: 0xb0360e8d, 0.0.0.0/0.0.0.0:45892] timeout=300.0>) 2021-05-03 00:04:11,200 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173371 type=client open_count=1 channel=[id: 0x05024b6c, 0.0.0.0/0.0.0.0:46363] timeout=300.0>) 2021-05-03 00:04:13,221 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173372 type=client open_count=1 channel=[id: 0x0fd4da8d, 0.0.0.0/0.0.0.0:35593] timeout=300.0>) 2021-05-03 00:04:15,249 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173373 type=client open_count=1 channel=[id: 0x71594a10, 0.0.0.0/0.0.0.0:51353] timeout=300.0>) 2021-05-03 00:04:17,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173374 type=client open_count=1 channel=[id: 0xf3cb1da4, 0.0.0.0/0.0.0.0:45129] timeout=300.0>) 2021-05-03 00:04:19,289 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173375 type=client open_count=1 channel=[id: 0xd74aa196, 0.0.0.0/0.0.0.0:59823] timeout=300.0>) 2021-05-03 00:04:21,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173376 type=client open_count=1 channel=[id: 0x2a8203a2, 0.0.0.0/0.0.0.0:59453] timeout=300.0>) 2021-05-03 00:04:23,338 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173377 type=client open_count=1 channel=[id: 0xd250e543, 0.0.0.0/0.0.0.0:53215] timeout=300.0>) 2021-05-03 00:04:25,375 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173378 type=client open_count=1 channel=[id: 0x9a3e274b, 0.0.0.0/0.0.0.0:39485] timeout=300.0>) 2021-05-03 00:04:27,415 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173379 type=client open_count=1 channel=[id: 0xf099d94d, 0.0.0.0/0.0.0.0:59650] timeout=300.0>) 2021-05-03 00:04:34,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17337a type=client open_count=1 channel=[id: 0x16812bef, 0.0.0.0/0.0.0.0:35697] timeout=300.0>) 2021-05-03 00:04:36,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17337b type=client open_count=1 channel=[id: 0x6b5ec127, 0.0.0.0/0.0.0.0:46614] timeout=300.0>) 2021-05-03 00:04:38,901 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17337c type=client open_count=1 channel=[id: 0xc8a0baed, 0.0.0.0/0.0.0.0:43581] timeout=300.0>) 2021-05-03 00:04:40,927 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17337d type=client open_count=1 channel=[id: 0x87a0a719, 0.0.0.0/0.0.0.0:58684] timeout=300.0>) 2021-05-03 00:04:42,944 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17337e type=client open_count=1 channel=[id: 0x4f679528, 0.0.0.0/0.0.0.0:37925] timeout=300.0>) 2021-05-03 00:04:44,980 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17337f type=client open_count=1 channel=[id: 0xd8d0a7b6, 0.0.0.0/0.0.0.0:37845] timeout=300.0>) 2021-05-03 00:04:47,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173380 type=client open_count=1 channel=[id: 0xe9a19ede, 0.0.0.0/0.0.0.0:42663] timeout=300.0>) 2021-05-03 00:04:49,016 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173381 type=client open_count=1 channel=[id: 0xfe9f40df, 0.0.0.0/0.0.0.0:37345] timeout=300.0>) 2021-05-03 00:04:51,039 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173382 type=client open_count=1 channel=[id: 0xdfe357f7, 0.0.0.0/0.0.0.0:49215] timeout=300.0>) 2021-05-03 00:04:58,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173383 type=client open_count=1 channel=[id: 0xda3f07b8, 0.0.0.0/0.0.0.0:41915] timeout=300.0>) 2021-05-03 00:05:00,653 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173384 type=client open_count=1 channel=[id: 0x28f1506f, 0.0.0.0/0.0.0.0:49000] timeout=300.0>) 2021-05-03 00:05:02,674 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173385 type=client open_count=1 channel=[id: 0x0106c5d5, 0.0.0.0/0.0.0.0:49641] timeout=300.0>) 2021-05-03 00:05:04,701 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173386 type=client open_count=1 channel=[id: 0x47c02e1b, 0.0.0.0/0.0.0.0:38311] timeout=300.0>) 2021-05-03 00:05:06,727 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173387 type=client open_count=1 channel=[id: 0x98d11dfe, 0.0.0.0/0.0.0.0:44035] timeout=300.0>) 2021-05-03 00:05:08,750 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173388 type=client open_count=1 channel=[id: 0xd642ff90, 0.0.0.0/0.0.0.0:60950] timeout=300.0>) 2021-05-03 00:05:10,770 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173389 type=client open_count=1 channel=[id: 0x6c52724a, 0.0.0.0/0.0.0.0:43523] timeout=300.0>) 2021-05-03 00:05:12,792 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17338a type=client open_count=1 channel=[id: 0xa0d87754, 0.0.0.0/0.0.0.0:55644] timeout=300.0>) 2021-05-03 00:05:14,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17338b type=client open_count=1 channel=[id: 0xbe649b38, 0.0.0.0/0.0.0.0:53194] timeout=300.0>) 2021-05-03 00:05:16,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17338c type=client open_count=1 channel=[id: 0x97516979, 0.0.0.0/0.0.0.0:33921] timeout=300.0>) 2021-05-03 00:05:24,654 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17338d type=client open_count=1 channel=[id: 0x3b5bd4d5, 0.0.0.0/0.0.0.0:41849] timeout=300.0>) 2021-05-03 00:05:26,677 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17338e type=client open_count=1 channel=[id: 0x41009e0f, 0.0.0.0/0.0.0.0:41373] timeout=300.0>) 2021-05-03 00:05:28,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17338f type=client open_count=1 channel=[id: 0x9c32e932, 0.0.0.0/0.0.0.0:47435] timeout=300.0>) 2021-05-03 00:05:30,723 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173390 type=client open_count=1 channel=[id: 0x168a756b, 0.0.0.0/0.0.0.0:60159] timeout=300.0>) 2021-05-03 00:05:32,744 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173391 type=client open_count=1 channel=[id: 0x46475bb1, 0.0.0.0/0.0.0.0:58307] timeout=300.0>) 2021-05-03 00:05:34,767 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173392 type=client open_count=1 channel=[id: 0x1409d827, 0.0.0.0/0.0.0.0:37663] timeout=300.0>) 2021-05-03 00:05:36,789 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173393 type=client open_count=1 channel=[id: 0x8d2e06bb, 0.0.0.0/0.0.0.0:50902] timeout=300.0>) 2021-05-03 00:05:38,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173394 type=client open_count=1 channel=[id: 0x9045f57d, 0.0.0.0/0.0.0.0:58399] timeout=300.0>) 2021-05-03 00:05:40,836 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173395 type=client open_count=1 channel=[id: 0xf5096258, 0.0.0.0/0.0.0.0:36680] timeout=300.0>) 2021-05-03 00:05:45,612 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173396 type=client open_count=1 channel=[id: 0xf2705fc9, 0.0.0.0/0.0.0.0:35877] timeout=300.0>) 2021-05-03 00:05:47,637 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173397 type=client open_count=1 channel=[id: 0xfdf04678, 0.0.0.0/0.0.0.0:40433] timeout=300.0>) 2021-05-03 00:05:49,658 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173398 type=client open_count=1 channel=[id: 0xdb173bac, 0.0.0.0/0.0.0.0:48359] timeout=300.0>) 2021-05-03 00:05:51,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173399 type=client open_count=1 channel=[id: 0x6c9ddbc4, 0.0.0.0/0.0.0.0:33941] timeout=300.0>) 2021-05-03 00:05:53,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17339a type=client open_count=1 channel=[id: 0xa2cbd95e, 0.0.0.0/0.0.0.0:45155] timeout=300.0>) 2021-05-03 00:05:55,951 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 00:06:12,522 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 00:06:26,864 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 00:06:26,890 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 00:06:26,934 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 00:07:00,470 | test | INFO | pool-8-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 8.47628657921 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:07:00,500 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:07:05,552 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:07:15,676 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:07:15,713 | test | INFO | pool-8-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.242000103 sec 2021-05-03 00:07:16,007 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 00:07:16,253 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 00:07:16,453 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | liH4HqxTx2FD4K3DrtdL-16-62000 | couchbase | 2 | none | 0 | 0 | 10825498624 | 0 | 0 | +-------------------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-03 00:07:16,454 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 00:07:18,654 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 00:07:18,663 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 00:07:46,243 | test | INFO | pool-8-thread-9 | [table_view:display:72] Ops trend for bucket 'liH4HqxTx2FD4K3DrtdL-16-62000' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 117.765 | ..........................*...............................................**..........................*X | 434.965 | | 422.100 | ................................................................................................*X | 422.100 | | 415.700 | ..............................................................................................X | 415.700 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 00:07:46,328 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 22.3864836325 | 3693387776 | 2967638016 | 0 / 3909087232 | 1951 / 3920 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 21.7516339869 | 3693387776 | 3050332160 | 0 / 3909087232 | 2449 / 4844 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 20.5619412516 | 6078930944 | 5222526976 | 0 / 3758092288 | 2051 / 4048 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 26.428385077 | 4201836544 | 3554365440 | 0 / 3758092288 | 2097 / 4212 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 00:07:52,247 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 00:08:14,252 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | liH4HqxTx2FD4K3DrtdL-16-62000 | couchbase | 2 | none | 0 | 10000 | 10825498624 | 103467920 | 179832410 | +-------------------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 00:08:14,253 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 00:08:26,459 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 00:08:48,016 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 0 2021-05-03 00:08:57,710 | test | INFO | pool-8-thread-23 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.09299993515 seconds to complete 2021-05-03 00:08:57,716 | test | INFO | pool-8-thread-23 | [common_lib:sleep:22] Sleep 8.09299993515 seconds. Reason: Wait after rebalance complete 2021-05-03 00:09:06,006 | test | INFO | pool-8-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 8.82129277567 | --- OUT ---> | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 17.2911392405 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 3.11088810838 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 12.4393358876 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:09:06,029 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:09:11,111 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 16.93 2021-05-03 00:09:16,180 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 40.88 2021-05-03 00:09:21,256 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 63.42 2021-05-03 00:09:26,328 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 87.79 Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17349b type=client open_count=1 channel=[id: 0x38b9d27f, 0.0.0.0/0.0.0.0:33866] timeout=300.0>) 2021-05-03 00:09:31,371 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17349c type=client open_count=1 channel=[id: 0x97b6c909, 0.0.0.0/0.0.0.0:47807] timeout=300.0>) 2021-05-03 00:09:33,388 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17349d type=client open_count=1 channel=[id: 0x90ddf646, 0.0.0.0/0.0.0.0:40961] timeout=300.0>) 2021-05-03 00:09:35,401 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17349e type=client open_count=1 channel=[id: 0x7a5250f5, 0.0.0.0/0.0.0.0:60158] timeout=300.0>) 2021-05-03 00:09:37,414 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17349f type=client open_count=1 channel=[id: 0xcf9c66d7, 0.0.0.0/0.0.0.0:48945] timeout=300.0>) 2021-05-03 00:09:39,437 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused 2021-05-03 00:09:46,555 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:09:46,579 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:748] GET http://172.23.98.196:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.98.196:8091/pools/default with status 0: unknown pool 2021-05-03 00:09:46,657 | test | INFO | pool-8-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 40.6489999294 sec 2021-05-03 00:09:46,765 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.104.186 2021-05-03 00:09:46,765 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.98.196 2021-05-03 00:09:58,502 | test | INFO | pool-8-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.38042381433 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 4.09342039176 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 4.44726810673 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:09:58,528 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:10:03,605 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:10:08,687 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 13.44 2021-05-03 00:10:13,770 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 34.89 2021-05-03 00:10:18,867 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 55.01 2021-05-03 00:10:23,982 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 75.08 2021-05-03 00:10:29,088 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 94.65 2021-05-03 00:10:39,206 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:10:39,253 | test | INFO | pool-8-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 40.7509999275 sec 2021-05-03 00:10:39,253 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 1 2021-05-03 00:10:49,003 | test | INFO | pool-8-thread-26 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.12699985504 seconds to complete 2021-05-03 00:10:49,009 | test | INFO | pool-8-thread-26 | [common_lib:sleep:22] Sleep 8.12699985504 seconds. Reason: Wait after rebalance complete 2021-05-03 00:10:57,331 | test | INFO | pool-8-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 7.51018329939 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 2.57640818388 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 2.48493975904 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 3.59584704989 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:10:57,355 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:11:02,430 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 14.96 2021-05-03 00:11:07,503 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 37.85 2021-05-03 00:11:12,585 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 60.99 2021-05-03 00:11:17,670 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 84.1 2021-05-03 00:11:27,766 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1734a0 type=client open_count=1 channel=[id: 0xadb9d1ff, 0.0.0.0/0.0.0.0:39265] timeout=300.0>) 2021-05-03 00:11:27,808 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1734a1 type=client open_count=1 channel=[id: 0x87b00f2b, 0.0.0.0/0.0.0.0:35423] timeout=300.0>) 2021-05-03 00:11:29,821 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1734a2 type=client open_count=1 channel=[id: 0x563b8cc1, 0.0.0.0/0.0.0.0:57081] timeout=300.0>) 2021-05-03 00:11:31,832 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 00:11:33,904 | test | INFO | pool-8-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 36.5730001926 sec 2021-05-03 00:11:33,980 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-03 00:11:35,062 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.121.10 2021-05-03 00:11:35,063 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.104.186 2021-05-03 00:11:45,938 | test | INFO | pool-8-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 5.78093306288 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 5.18200202224 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 5.83142347848 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:11:45,967 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:11:51,045 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:11:56,118 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:12:01,207 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 13.05 2021-05-03 00:12:06,322 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.09 2021-05-03 00:12:11,418 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 50.23 2021-05-03 00:12:16,532 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 71.83 2021-05-03 00:12:21,614 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 91.52 2021-05-03 00:12:31,724 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:12:31,763 | test | INFO | pool-8-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 45.8250000477 sec 2021-05-03 00:12:31,765 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 2 2021-05-03 00:12:41,450 | test | INFO | pool-8-thread-29 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.11899995804 seconds to complete 2021-05-03 00:12:41,456 | test | INFO | pool-8-thread-29 | [common_lib:sleep:22] Sleep 8.11899995804 seconds. Reason: Wait after rebalance complete 2021-05-03 00:12:49,776 | test | INFO | pool-8-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 2.40323804705 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 4.5179202423 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 8.01708972104 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 1.97019449356 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:12:49,802 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:12:54,878 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 14.01 2021-05-03 00:12:59,983 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 34.17 2021-05-03 00:13:05,058 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 57.19 2021-05-03 00:13:10,335 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 81.93 2021-05-03 00:13:15,645 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 00:13:25,729 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1734a3 type=client open_count=1 channel=[id: 0x659202a5, 0.0.0.0/0.0.0.0:43453] timeout=300.0>) 2021-05-03 00:13:25,767 | test | ERROR | pool-8-thread-30 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1734a4 type=client open_count=1 channel=[id: 0x3f761067, 0.0.0.0/0.0.0.0:38287] timeout=300.0>) 2021-05-03 00:13:27,822 | test | ERROR | pool-8-thread-30 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 00:13:29,960 | test | INFO | pool-8-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 40.1840000153 sec 2021-05-03 00:13:30,071 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.104.186 2021-05-03 00:13:30,072 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.121.10 2021-05-03 00:13:41,612 | test | INFO | pool-8-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 4.35443037975 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 5.53449583017 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 5.86319218241 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:13:41,641 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:13:46,704 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:13:51,778 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:13:56,881 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 14.72 2021-05-03 00:14:01,984 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.47 2021-05-03 00:14:07,075 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 52.69 2021-05-03 00:14:12,177 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 75.27 2021-05-03 00:14:17,306 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 92.29 2021-05-03 00:14:27,404 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:14:27,441 | test | INFO | pool-8-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 45.8289999962 sec 2021-05-03 00:14:27,443 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 3 2021-05-03 00:14:37,049 | test | INFO | pool-8-thread-4 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.09800004959 seconds to complete 2021-05-03 00:14:37,055 | test | INFO | pool-8-thread-4 | [common_lib:sleep:22] Sleep 8.09800004959 seconds. Reason: Wait after rebalance complete 2021-05-03 00:14:45,348 | test | INFO | pool-8-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 16.3151207116 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.30057949106 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 1.70298021538 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 3.75634517766 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:14:45,377 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:14:50,464 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 17.82 2021-05-03 00:14:55,559 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 42.0 2021-05-03 00:15:00,635 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 65.88 2021-05-03 00:15:05,710 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 94.6 2021-05-03 00:15:15,802 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1734a5 type=client open_count=1 channel=[id: 0xd7ce95b4, 0.0.0.0/0.0.0.0:39955] timeout=300.0>) 2021-05-03 00:15:15,842 | test | ERROR | pool-8-thread-1 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1734a6 type=client open_count=1 channel=[id: 0x60a5bc84, 0.0.0.0/0.0.0.0:58986] timeout=300.0>) 2021-05-03 00:15:17,855 | test | ERROR | pool-8-thread-1 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 00:15:19,934 | test | INFO | pool-8-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 34.5869998932 sec 2021-05-03 00:15:20,065 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.121.10 2021-05-03 00:15:20,066 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.104.186 2021-05-03 00:15:31,213 | test | INFO | pool-8-thread-2 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.25317742755 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 7.22222222222 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.39893211289 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:15:31,240 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:15:36,312 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:15:41,407 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:15:46,494 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 9.84 2021-05-03 00:15:51,617 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 27.62 2021-05-03 00:15:56,706 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 47.45 2021-05-03 00:16:01,790 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 65.82 2021-05-03 00:16:06,878 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 87.77 2021-05-03 00:16:16,987 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:16:17,028 | test | INFO | pool-8-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 45.8140001297 sec 2021-05-03 00:16:18,332 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: liH4HqxTx2FD4K3DrtdL-16-62000, Active Resident ratio(DGM): 100% 2021-05-03 00:16:18,334 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: liH4HqxTx2FD4K3DrtdL-16-62000, Replica Resident ratio(DGM): 100% 2021-05-03 00:16:38,709 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 00:16:52,167 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1549999714 seconds to complete 2021-05-03 00:16:52,186 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 778.104s OK During the test, Remote Connections: 82, Disconnections: 82 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 1 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-02_21-20-18/test_6 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=3,nodes_swap=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=3,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 2h 56m 54s 2 actionable tasks: 1 executed, 1 up-to-date + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_rebalance_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'cluster_name': 'win10-bucket-ops', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops.ini', 'num_nodes': 5, 'quota_percent': '95', 'rerun': 'False', 'spec': 'collections_rebalance_crud_on_collections'} Only cases in GROUPs 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' will be executed Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_1', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 1, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 00:17:20,272 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 00:17:21,655 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 00:17:23,197 | test | ERROR | MainThread | [rest_client:_http_request:748] GET http://172.23.98.196:8091/pools/default/ body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-05-03 00:17:23,900 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xf904dfc0, 0.0.0.0/0.0.0.0:34747] timeout=300.0>) 2021-05-03 00:17:29,348 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0xe954d61d, 0.0.0.0/0.0.0.0:42359] timeout=300.0>) 2021-05-03 00:17:32,388 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0xcb9480f9, 0.0.0.0/0.0.0.0:37527] timeout=300.0>) 2021-05-03 00:17:34,444 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x1406b4b4, 0.0.0.0/0.0.0.0:37119] timeout=300.0>) 2021-05-03 00:17:36,494 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x3a45fc35, 0.0.0.0/0.0.0.0:52568] timeout=300.0>) 2021-05-03 00:17:38,539 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x50379399, 0.0.0.0/0.0.0.0:60076] timeout=300.0>) 2021-05-03 00:17:40,579 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x07cadd32, 0.0.0.0/0.0.0.0:44257] timeout=300.0>) 2021-05-03 00:17:42,624 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x6e3ea8bb, 0.0.0.0/0.0.0.0:47018] timeout=300.0>) 2021-05-03 00:17:44,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x275f1dad, 0.0.0.0/0.0.0.0:59031] timeout=300.0>) 2021-05-03 00:17:46,707 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x032673c1, 0.0.0.0/0.0.0.0:33912] timeout=300.0>) 2021-05-03 00:17:54,411 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0xdb526e99, 0.0.0.0/0.0.0.0:38321] timeout=300.0>) 2021-05-03 00:17:56,450 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x3ab808ff, 0.0.0.0/0.0.0.0:40219] timeout=300.0>) 2021-05-03 00:17:58,490 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0xb4325305, 0.0.0.0/0.0.0.0:37693] timeout=300.0>) 2021-05-03 00:18:00,535 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xc33e4743, 0.0.0.0/0.0.0.0:45623] timeout=300.0>) 2021-05-03 00:18:02,572 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0xa4e2c3ac, 0.0.0.0/0.0.0.0:46459] timeout=300.0>) 2021-05-03 00:18:04,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0xa61a0b77, 0.0.0.0/0.0.0.0:55131] timeout=300.0>) 2021-05-03 00:18:06,647 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x3c970157, 0.0.0.0/0.0.0.0:43861] timeout=300.0>) 2021-05-03 00:18:08,734 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0x983c4e93, 0.0.0.0/0.0.0.0:34359] timeout=300.0>) 2021-05-03 00:18:10,769 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x74b41bac, 0.0.0.0/0.0.0.0:44565] timeout=300.0>) 2021-05-03 00:18:18,456 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x0c4f423c, 0.0.0.0/0.0.0.0:44713] timeout=300.0>) 2021-05-03 00:18:20,490 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0xe80b1496, 0.0.0.0/0.0.0.0:43169] timeout=300.0>) 2021-05-03 00:18:22,532 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x332a0e6a, 0.0.0.0/0.0.0.0:58955] timeout=300.0>) 2021-05-03 00:18:24,565 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0xbd957eff, 0.0.0.0/0.0.0.0:33710] timeout=300.0>) 2021-05-03 00:18:26,595 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x9526d857, 0.0.0.0/0.0.0.0:33712] timeout=300.0>) 2021-05-03 00:18:28,625 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0x3e9ca312, 0.0.0.0/0.0.0.0:35476] timeout=300.0>) 2021-05-03 00:18:30,661 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0xcb4565db, 0.0.0.0/0.0.0.0:41231] timeout=300.0>) 2021-05-03 00:18:32,693 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0x66aea054, 0.0.0.0/0.0.0.0:58655] timeout=300.0>) 2021-05-03 00:18:34,734 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0xbfa34cea, 0.0.0.0/0.0.0.0:46279] timeout=300.0>) 2021-05-03 00:18:36,772 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0xd3ed01f5, 0.0.0.0/0.0.0.0:53392] timeout=300.0>) 2021-05-03 00:18:38,802 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2a type=client open_count=1 channel=[id: 0xf10bae0d, 0.0.0.0/0.0.0.0:38378] timeout=300.0>) 2021-05-03 00:18:46,970 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2b type=client open_count=1 channel=[id: 0x881bb7c7, 0.0.0.0/0.0.0.0:35470] timeout=300.0>) 2021-05-03 00:18:49,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2c type=client open_count=1 channel=[id: 0x6ca7b81e, 0.0.0.0/0.0.0.0:46067] timeout=300.0>) 2021-05-03 00:18:51,043 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2d type=client open_count=1 channel=[id: 0x0973d550, 0.0.0.0/0.0.0.0:34189] timeout=300.0>) 2021-05-03 00:18:53,075 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2e type=client open_count=1 channel=[id: 0xdcd9cca9, 0.0.0.0/0.0.0.0:55038] timeout=300.0>) 2021-05-03 00:18:55,117 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2f type=client open_count=1 channel=[id: 0x8c746f3a, 0.0.0.0/0.0.0.0:51655] timeout=300.0>) 2021-05-03 00:18:57,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x30 type=client open_count=1 channel=[id: 0x7c20e5d0, 0.0.0.0/0.0.0.0:60506] timeout=300.0>) 2021-05-03 00:18:59,293 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x31 type=client open_count=1 channel=[id: 0x37d70ee8, 0.0.0.0/0.0.0.0:47155] timeout=300.0>) 2021-05-03 00:19:01,322 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x32 type=client open_count=1 channel=[id: 0x049fbab4, 0.0.0.0/0.0.0.0:36361] timeout=300.0>) 2021-05-03 00:19:03,351 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x33 type=client open_count=1 channel=[id: 0x12159331, 0.0.0.0/0.0.0.0:33810] timeout=300.0>) 2021-05-03 00:19:08,392 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x34 type=client open_count=1 channel=[id: 0xdc1ea2df, 0.0.0.0/0.0.0.0:33805] timeout=300.0>) 2021-05-03 00:19:10,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x35 type=client open_count=1 channel=[id: 0xb86c6d66, 0.0.0.0/0.0.0.0:34704] timeout=300.0>) 2021-05-03 00:19:12,460 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x36 type=client open_count=1 channel=[id: 0xe76b47e2, 0.0.0.0/0.0.0.0:34863] timeout=300.0>) 2021-05-03 00:19:14,499 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x37 type=client open_count=1 channel=[id: 0x7541dec4, 0.0.0.0/0.0.0.0:47171] timeout=300.0>) 2021-05-03 00:19:16,526 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x38 type=client open_count=1 channel=[id: 0xfcb57df3, 0.0.0.0/0.0.0.0:57739] timeout=300.0>) 2021-05-03 00:19:18,553 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 00:19:35,575 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 00:19:49,835 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 00:19:49,868 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 00:19:49,960 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 00:20:11,243 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 8.49886849384 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:20:11,279 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:20:21,401 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:20:21,447 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2049999237 sec 2021-05-03 00:20:21,956 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 00:21:07,956 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 00:21:08,316 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 629145600 | 661969992 | 46431381 | | bucket2 | couchbase | 2 | none | 0 | 0 | 629145600 | 606445016 | 66850418 | | default | couchbase | 2 | none | 0 | 0 | 6291456000 | 792322168 | 28252766 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-03 00:21:08,318 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 00:22:21,250 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 00:22:21,332 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 00:43:23,440 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***.................................**.................................*..................................X | 31.517 | | 21.017 | .................................................................*X | 21.017 | | 10.517 | ................................X | 10.517 | | 0.000 | **********************............***.......X | 6.000 | | 4.000 | ...........X | 4.000 | | 2.000 | .....*X | 2.000 | | 0.000 | ********.*...**...**X | 2.500 | | 1.500 | ...**.............**.........**..X | 9.100 | | 8.598 | ..........................**.....*X | 10.098 | | 9.498 | .............................*.*X | 10.003 | | 6.503 | ...................*X | 6.503 | | 6.003 | ..................X | 6.003 | | 4.500 | .............*.....*X | 6.000 | | 5.000 | ..............*X | 5.000 | | 4.500 | .............X | 4.500 | | 2.500 | ......*X | 2.500 | | 1.999 | .....X | 1.999 | | 1.499 | ...*X | 1.499 | | 0.999 | ..*X | 0.999 | | 0.000 | **********************************.**..*.*X | 1.500 | | 1.000 | ..*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ***********************************************************************************************....................*.....................**....................*X | 19.500 | | 13.000 | ........................................X | 13.000 | | 6.500 | ...................X | 6.500 | | 0.000 | ******************************************************........**...............................**....................................................*X | 28.697 | | 23.201 | ........................................................................*X | 23.201 | | 14.701 | .............................................*X | 14.701 | | 9.001 | ...........................X | 9.001 | | 5.000 | ..............*X | 5.000 | | 2.500 | ......X | 2.500 | | 1.500 | ...**........**.....................*...........*.....*X | 15.499 | | 8.999 | ...........................X | 8.999 | | 4.000 | ...........*X | 4.000 | | 0.000 | ******************************......**......**..........................**.........**...**...........*.....*X | 21.000 | | 14.500 | .............................................X | 14.500 | | 8.000 | ........................*X | 8.000 | | 4.500 | .............**...........**X | 8.000 | | 5.000 | ..............X | 5.000 | | 3.000 | ........*....**.....................**..............X | 15.302 | | 15.200 | ...............................................X | 15.200 | | 8.698 | ..........................**X | 8.698 | | 4.001 | ...........**.......*X | 6.001 | | 4.001 | ...........X | 4.001 | | 2.000 | .....*X | 2.000 | | 0.000 | ****.**..*.*X | 1.500 | | 1.499 | ...***X | 1.499 | | 1.000 | ..*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | *******************.**..**.*.......*............**................*...**X | 13.501 | | 12.001 | .....................................X | 12.001 | | 9.001 | ...........................X | 9.001 | | 8.000 | ........................*X | 8.000 | | 6.000 | ..................*X | 6.000 | | 4.000 | ...........X | 4.000 | | 1.500 | ...X | 1.500 | | 0.500 | *X | 0.500 | | 0.000 | ***.**......*..........**........***X | 8.002 | | 7.502 | ......................**X | 7.502 | | 4.500 | .............*X | 4.500 | | 2.000 | .....*X | 2.000 | | 0.500 | *..**.X | 1.500 | | 1.000 | ..X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ***************************.**..*.*X | 1.500 | | 1.000 | ..*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | **....**.....*.....X | 4.499 | | 2.999 | ........*X | 2.999 | | 1.499 | ...X | 1.499 | | 0.000 | **....**.....*......*X | 5.000 | | 4.500 | .............X | 4.500 | | 4.000 | ...........X | 4.000 | | 2.500 | ......*X | 2.500 | | 1.500 | ...*X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ****.*..*.*X | 1.501 | | 1.000 | ..X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ***.*..**....*..**..*X | 3.500 | | 2.002 | .....X | 2.002 | | 1.000 | ..X | 1.000 | | 0.000 | **********************************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 00:43:24,012 | test | INFO | pool-2-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.................**..................................*...................................*..............*X | 989.455 | | 914.455 | ...........................................................................................*X | 914.455 | | 810.700 | ................................................................................X | 810.700 | | 743.400 | ..........................................................................*X | 743.400 | | 676.000 | ...................................................................X | 676.000 | | 637.400 | ...............................................................X | 637.400 | | 586.500 | ..........................................................**.*..***X | 621.374 | | 554.808 | .......................................................****X | 556.864 | | 550.864 | ......................................................*X | 550.864 | | 536.964 | .....................................................*X | 536.964 | | 530.735 | ....................................................X | 530.735 | | 521.635 | ...................................................*X | 521.635 | | 510.635 | ..................................................*X | 510.635 | | 480.300 | ...............................................*X | 480.300 | | 446.200 | ............................................*X | 446.200 | | 380.700 | .....................................*X | 380.700 | | 352.400 | ..................................X | 352.400 | | 340.400 | .................................**...*...***X | 404.758 | | 395.436 | ......................................**.*X | 397.536 | | 364.436 | ...................................**X | 365.200 | | 354.600 | ..................................**...*...*X | 409.564 | | 407.064 | ........................................*X | 407.064 | | 321.081 | ...............................**.**..*.......*X | 420.000 | | 406.176 | ........................................**X | 411.276 | | 393.876 | ......................................**...***.*..**..**X | 473.851 | | 447.762 | ............................................*X | 447.762 | | 386.550 | ......................................X | 386.550 | | 353.050 | ..................................X | 353.050 | | 342.353 | .................................**....*..*X | 396.395 | | 388.441 | ......................................*X | 388.441 | | 356.546 | ...................................X | 356.546 | | 354.946 | ..................................****..X | 374.455 | | 373.055 | ....................................*..***X | 390.000 | | 384.400 | .....................................X | 384.400 | | 381.700 | .....................................**..*.....**..*X | 472.000 | | 449.800 | ............................................**..**.X | 481.080 | | 466.780 | ..............................................*X | 466.780 | | 439.101 | ...........................................*X | 439.101 | | 426.501 | ..........................................*..*X | 450.901 | | 442.132 | ...........................................X | 442.132 | | 438.332 | ...........................................X | 438.332 | | 400.832 | .......................................**.**.*..*X | 442.472 | | 427.236 | ..........................................X | 427.236 | | 389.236 | ......................................X | 389.236 | | 348.736 | ..................................*X | 348.736 | | 319.650 | ...............................*X | 319.650 | | 305.750 | .............................X | 305.750 | | 293.750 | ............................*X | 293.750 | | 288.325 | ............................*X | 293.825 | | 292.525 | ............................**...**.*...*X | 360.199 | | 353.900 | ..................................*.X | 361.700 | | 356.800 | ...................................**.**....*.......**.......*.X | 557.841 | | 541.941 | .....................................................*X | 541.941 | | 478.300 | ...............................................*X | 478.300 | | 449.300 | ............................................X | 449.300 | | 423.400 | .........................................*X | 423.400 | | 421.017 | .........................................*.*.**.*X | 449.488 | | 442.873 | ...........................................*X | 443.373 | | 420.485 | .........................................**.***.......*X | 497.723 | | 461.400 | .............................................X | 461.400 | | 403.000 | .......................................*X | 403.000 | | 300.800 | .............................*X | 300.800 | | 296.900 | .............................*.**..X | 329.700 | | 328.842 | ................................X | 328.842 | | 324.842 | ...............................**...*X | 354.842 | | 352.454 | ..................................*.*X | 363.654 | | 347.054 | ..................................*X | 347.054 | | 327.000 | ................................**X | 327.000 | | 316.200 | ..............................X | 316.200 | | 308.100 | ..............................***..**...*....**...*X | 427.229 | | 410.729 | ........................................X | 410.729 | | 390.629 | ......................................X | 390.629 | | 368.400 | ....................................**...*....**.....*X | 489.536 | | 449.755 | ............................................**X | 449.755 | | 407.255 | ........................................X | 407.255 | | 401.155 | .......................................X | 401.155 | | 396.600 | .......................................*X | 396.600 | | 377.700 | .....................................*X | 377.700 | | 363.000 | ...................................***.......X | 426.600 | | 408.400 | ........................................*...*X | 444.400 | | 378.500 | .....................................**X | 382.600 | | 340.297 | .................................*X | 340.297 | | 339.197 | .................................**.X | 346.497 | | 340.793 | .................................**..X | 364.193 | | 339.893 | .................................*.*X | 352.000 | | 335.698 | ................................**..X | 355.698 | | 344.098 | .................................X | 344.098 | | 333.893 | ................................**.**.*....**.*..*X | 416.500 | | 407.600 | ........................................**.*X | 422.900 | | 421.300 | .........................................*X | 421.300 | | 384.200 | .....................................*X | 384.200 | | 355.600 | ..................................X | 355.600 | | 331.368 | ................................*X | 331.368 | | 317.368 | ...............................X | 317.368 | | 304.368 | .............................**.**..**X | 335.734 | | 326.500 | ...............................X | 326.500 | | 307.900 | ..............................X | 307.900 | | 307.100 | ..............................****.**....***........**X | 444.730 | | 437.337 | ...........................................X | 437.337 | | 430.135 | ..........................................*..**..*X | 465.173 | | 458.248 | .............................................***..X | 478.366 | | 476.399 | ...............................................X | 476.399 | | 473.970 | ..............................................*X | 473.970 | | 420.364 | .........................................*X | 420.364 | | 333.720 | ................................**..**.....*.......**....**.X | 522.969 | | 474.169 | ..............................................*X | 474.169 | | 425.311 | .........................................*X | 425.311 | | 344.358 | .................................**X | 344.358 | | 323.246 | ...............................X | 323.246 | | 305.745 | .............................X | 305.745 | | 291.445 | ............................*X | 291.445 | | 285.945 | ...........................**...X | 310.600 | | 307.500 | ..............................*..***........***.*X | 421.300 | | 380.200 | .....................................*X | 380.200 | | 374.263 | ....................................*X | 374.263 | | 312.563 | ..............................X | 312.563 | | 279.363 | ...........................*X | 279.363 | | 236.324 | ......................**..............***.......X | 438.786 | | 415.786 | .........................................X | 415.786 | | 409.122 | ........................................*X | 409.122 | | 390.580 | ......................................*X | 390.580 | | 390.430 | ......................................X | 390.430 | | 369.600 | ....................................X | 369.600 | | 331.373 | ................................*X | 331.373 | | 324.769 | ...............................*X | 324.769 | | 321.547 | ...............................*...*X | 347.747 | | 341.600 | .................................*...*..**.....**...**X | 474.224 | | 459.524 | .............................................X | 459.524 | | 444.924 | ...........................................*.*X | 448.831 | | 411.531 | ........................................*X | 411.531 | | 390.331 | ......................................X | 390.331 | | 334.707 | ................................*X | 334.707 | | 327.307 | ................................X | 327.307 | | 299.207 | .............................*...**...*.....*X | 412.400 | | 408.700 | ........................................**X | 409.000 | | 391.000 | ......................................*X | 391.000 | | 380.800 | .....................................X | 380.800 | | 377.700 | .....................................X | 377.700 | | 368.764 | ....................................**X | 368.764 | | 348.864 | ..................................X | 348.864 | | 328.664 | ................................*.**....**..X | 401.938 | | 400.300 | .......................................*..**....**.....X | 511.500 | | 494.400 | ................................................*X | 494.400 | | 454.700 | ............................................X | 454.700 | | 404.200 | .......................................*X | 404.200 | | 357.500 | ...................................*X | 357.500 | | 298.000 | .............................X | 298.000 | | 240.360 | .......................***...**......***...X | 357.576 | | 342.800 | .................................**..*X | 359.300 | | 343.622 | .................................*X | 344.422 | | 342.122 | .................................**..**X | 358.289 | | 343.589 | .................................*X | 343.589 | | 329.100 | ................................**.....**..*...**.**.X | 453.621 | | 451.421 | ............................................*X | 451.421 | | 443.444 | ...........................................X | 443.444 | | 426.044 | ..........................................X | 426.044 | | 390.944 | ......................................*X | 390.944 | | 373.024 | ....................................*X | 373.024 | | 316.124 | ..............................X | 316.124 | | 304.424 | .............................*X | 304.424 | | 289.421 | ............................**..*X | 310.421 | | 305.700 | .............................**.*....**...**.X | 386.260 | | 368.760 | ....................................X | 368.760 | | 341.031 | .................................**.*.***X | 365.499 | | 362.899 | ...................................*.**......*....*....*X | 509.500 | | 503.600 | .................................................*X | 503.600 | | 503.200 | .................................................X | 503.200 | | 467.800 | ..............................................X | 467.800 | | 431.800 | ..........................................*X | 431.800 | | 378.292 | .....................................X | 378.292 | | 338.892 | .................................*X | 338.892 | | 321.092 | ...............................**.X | 335.623 | | 331.923 | ................................*...*X | 356.223 | | 334.700 | ................................*.....*X | 384.600 | | 363.400 | ...................................**.....X | 415.400 | | 378.209 | .....................................**X | 383.309 | | 325.804 | ...............................X | 325.804 | | 321.478 | ...............................*X | 321.478 | | 310.578 | ..............................*X | 310.578 | | 308.987 | ..............................X | 308.987 | | 298.004 | .............................X | 298.004 | | 288.174 | ............................***.......**......**....*.*X | 466.755 | | 442.806 | ...........................................*X | 442.806 | | 425.904 | ..........................................**X | 425.904 | | 412.378 | ........................................X | 412.378 | | 371.396 | ....................................X | 371.396 | | 362.370 | ...................................*X | 362.370 | | 345.918 | .................................**....X | 380.027 | | 363.975 | ...................................**....**.....*X | 446.171 | | 443.871 | ...........................................*X | 443.871 | | 438.091 | ...........................................X | 438.091 | | 414.670 | ........................................*X | 414.670 | | 400.570 | .......................................*X | 400.570 | | 366.161 | ....................................X | 366.161 | +---------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 00:43:24,973 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **************************************************************.................**..................**..................X | 23.507 | | 16.507 | ....................................*X | 16.507 | | 8.507 | ..................X | 8.507 | | 0.500 | **..**..................**...........*X | 14.501 | | 14.495 | ...............................**X | 14.495 | | 7.994 | .................*X | 7.994 | | 4.000 | ........X | 4.000 | | 0.000 | ********************************************************************************************************************************************************************************************************************.*...**.........**..........*......*X | 12.999 | | 10.499 | ......................X | 10.499 | | 5.500 | ...........X | 5.500 | | 1.500 | ..*X | 1.500 | | 0.000 | *****....*.....*....*X | 6.000 | | 3.998 | ........*X | 3.998 | | 1.998 | ...X | 1.998 | | 0.000 | *******.**.....**........**......X | 9.001 | | 7.501 | ................*X | 7.501 | | 5.500 | ...........X | 5.500 | | 4.500 | .........*..***.....***.*......................*..............................**..............................*X | 44.000 | | 33.500 | ...........................................................................X | 33.500 | | 19.000 | ..........................................X | 19.000 | | 7.000 | ..............***......**X | 9.500 | | 6.000 | ............X | 6.000 | | 3.500 | ......*..**........*X | 7.501 | | 6.501 | .............*X | 6.501 | | 3.000 | .....*X | 3.000 | | 0.500 | *X | 0.500 | | 0.000 | ********************************************.*......**..........**...........*..*............**......*..............*X | 27.500 | | 22.500 | ..................................................*X | 22.500 | | 16.500 | ....................................X | 16.500 | | 6.500 | .............*X | 6.500 | | 2.500 | ....*X | 2.500 | | 0.000 | **...**.........**...............*...........**....X | 18.496 | | 16.303 | ....................................*X | 16.303 | | 12.803 | ............................*X | 12.803 | | 4.700 | .........**X | 4.700 | | 2.700 | .....X | 2.700 | | 2.699 | .....*...*X | 3.999 | | 3.499 | ......****...*X | 4.501 | | 3.000 | .....X | 3.000 | | 1.500 | ..*X | 1.500 | | 0.000 | ***********************.**.*.*X | 1.500 | | 1.000 | .X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | *******************.*.**...............*.............**...............*X | 19.994 | | 15.002 | .................................X | 15.002 | | 9.502 | ....................X | 9.502 | | 3.002 | .....*X | 3.002 | | 1.500 | ..X | 1.500 | | 0.500 | *X | 0.500 | | 0.000 | **.*........*........**............X | 12.987 | | 12.499 | ...........................*X | 12.499 | | 11.999 | ..........................*X | 11.999 | | 6.005 | ............X | 6.005 | | 3.000 | .....X | 3.000 | | 0.000 | ***********************************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 00:43:25,605 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 48.2931992591 | 3693387776 | 2499543040 | 0 / 3909087232 | 168497 / 337116 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 61.978197288 | 3693387776 | 2491228160 | 0 / 3909087232 | 168537 / 335859 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 47.197716658 | 4201836544 | 2973442048 | 0 / 3758092288 | 168264 / 337531 | 7.0.0-5085-enterprise | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 00:43:38,977 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 00:44:58,717 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 153096680 | 264144130 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 152988568 | 277183621 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 429989400 | 473329669 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-03 00:44:58,720 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 00:45:06,624 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_in 2021-05-03 00:45:06,625 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_in 2021-05-03 00:45:06,786 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 00:45:14,608 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 00:45:26,864 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 19.0452006094 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 13.9064475348 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 9.27388535032 | Cluster node | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 00:45:26,910 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:45:32,207 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 00:45:37,397 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.6 2021-05-03 00:45:43,270 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 5.43 2021-05-03 00:45:48,463 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 9.13 2021-05-03 00:45:53,634 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 12.81 2021-05-03 00:45:56,285 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-556 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:45:56,285 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-556 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:45:56,286 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-525 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-457 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-556 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-487 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:45:56,286 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-525 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-556 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-457 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-487 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:45:58,821 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 16.26 2021-05-03 00:46:01,575 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:01,578 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-721 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:01,582 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:01,582 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-721 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:03,956 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 20.56 2021-05-03 00:46:09,118 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 23.88 2021-05-03 00:46:09,526 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:09,529 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-870 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:09,538 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:09,540 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-870 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:14,250 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 26.78 2021-05-03 00:46:19,394 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.25 2021-05-03 00:46:22,845 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1017 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1187 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:22,848 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1017 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1187 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:22,848 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:Rx-12-680000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1027 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1017 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1161 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1187 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:22,848 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1027 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1017 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1161 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1187 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:26,726 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 00:46:28,622 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket1:aOi%-11-981000:j7WFG%yH8q0svVd-12-134000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-566 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:28,625 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket1:aOi%-11-981000:j7WFG%yH8q0svVd-12-134000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-566 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:31,861 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 38.34 2021-05-03 00:46:37,059 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.84 2021-05-03 00:46:38,938 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1065 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:38,940 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1065 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:38,941 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:B0jhs0rgkB8XDcdwgt-10-952000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1065 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:38,944 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:B0jhs0rgkB8XDcdwgt-10-952000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1065 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:39,341 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in bucket2:_default:LBQEG0_vOjmw-22-61000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:39,342 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in bucket2:scope1:6Hoo3BtK2Jk1aUs-22-71000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:39,341 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in bucket2:_default:S-22-53000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:39,341 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in bucket2:_default:QfwimWuabd4_EUMT-22-62000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:39,342 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in bucket2:_default:xsBthEWWh3dz-22-55000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:39,832 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in bucket2:_default:rJzOR5jvsvl6aG-22-56000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:42,308 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 45.11 2021-05-03 00:46:44,168 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:PRPR-11-370000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-827 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:44,170 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:PRPR-11-370000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-827 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-998 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:44,180 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-827 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:44,183 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-827 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-998 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:46,691 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in bucket2:k-11-624000:vsL2O%20MEPIR9a-11-779000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-801 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-991 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:46,694 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in bucket2:k-11-624000:vsL2O%20MEPIR9a-11-779000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-801 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-991 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:46,697 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-972 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:46,700 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-847 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-858 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-972 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:47,448 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 48.7 2021-05-03 00:46:48,198 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in bucket2:_default:in4-T_-22-67000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-0 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:48,198 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in bucket2:_default:-5E_pLP61DjO3a-22-62000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-0 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:48,203 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in bucket2:_default:EyRsc-22-51000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-0 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:50,867 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1312 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1282 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:50,868 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1312 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1282 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:50,871 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:B0jhs0rgkB8XDcdwgt-10-952000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1312 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1282 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:50,872 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:B0jhs0rgkB8XDcdwgt-10-952000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1312 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1257 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1282 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:52,729 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 53.01 2021-05-03 00:46:57,723 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1069 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:57,723 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1069 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:57,724 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1077 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1069 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:57,726 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1077 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1069 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:46:57,917 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 56.52 2021-05-03 00:47:01,128 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1524 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:01,130 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1524 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:03,073 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 59.01 2021-05-03 00:47:05,002 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:PRPR-11-370000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:05,003 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:PRPR-11-370000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1344 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:06,868 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in bucket2:k-11-624000:vsL2O%20MEPIR9a-11-779000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1215 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1385 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:06,871 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in bucket2:k-11-624000:vsL2O%20MEPIR9a-11-779000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1215 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1385 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:07,407 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:07,407 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:07,407 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:B0jhs0rgkB8XDcdwgt-10-952000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1562 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:07,408 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:07,410 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:07,411 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:B0jhs0rgkB8XDcdwgt-10-952000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1562 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:08,196 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 62.57 2021-05-03 00:47:12,072 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1647 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:12,073 | test | INFO | pool-2-thread-2 | [table_view:display:72] Keys failed in bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1632 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:13,423 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 66.3 2021-05-03 00:47:14,328 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1440 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:14,328 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in bucket2:k-11-624000:0dE-11-780000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1440 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:14,328 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:PRPR-11-370000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1440 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:14,331 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1440 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:14,331 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:PRPR-11-370000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1440 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:14,331 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in bucket2:k-11-624000:vsL2O%20MEPIR9a-11-779000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1440 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:14,332 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1440 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:14,332 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in bucket2:k-11-624000:vsL2O%20MEPIR9a-11-779000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1424 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1440 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:18,596 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 67.6 2021-05-03 00:47:22,604 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-691 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:23,763 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 71.75 2021-05-03 00:47:27,926 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-139 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-169 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:28,891 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 75.09 2021-05-03 00:47:32,658 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-58 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-139 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-169 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-144 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-141 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:32,726 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-629 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-636 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-633 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-620 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-691 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:33,125 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-828 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-859 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-701 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-742 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-850 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:34,026 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 77.45 2021-05-03 00:47:39,173 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 80.6 2021-05-03 00:47:42,230 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-828 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-845 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-789 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-742 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-850 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:44,506 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 84.11 2021-05-03 00:47:44,657 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:scope1:Wtj495VxWWqvls9hfxl-22-231000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-422 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-472 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:45,063 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2537 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2592 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:45,065 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2537 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2592 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2597 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:45,201 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-519 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-611 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-534 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-642 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-571 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:45,203 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-69 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-37 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-44 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-72 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-118 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-186 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-175 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-163 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:49,736 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 87.36 2021-05-03 00:47:51,865 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-218 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-207 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-306 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-359 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-327 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-388 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-322 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-345 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-246 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-254 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-397 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-211 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-296 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-250 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:52,453 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-319 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-327 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-289 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-246 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:54,974 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 90.53 2021-05-03 00:47:55,342 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:55,345 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2759 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:58,161 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-909 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-915 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-924 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-946 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-903 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-961 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:47:59,855 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-727 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-845 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-800 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-780 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-878 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-735 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-856 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-899 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-831 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-893 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:00,184 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 93.74 2021-05-03 00:48:01,375 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:01,377 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2759 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:01,417 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-915 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-959 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-946 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-968 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-903 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:02,967 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:02,970 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2882 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:05,397 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 97.23 2021-05-03 00:48:05,520 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-313 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-355 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-224 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-353 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-310 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-261 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-283 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-280 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:08,417 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:scope1:R74D-22-239000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-608 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-513 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-623 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-685 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-681 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:09,539 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-434 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-461 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:09,539 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:scope1:qU-22-233000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-434 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-461 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:10,513 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 99.81 2021-05-03 00:48:10,940 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:scope2:Ifr-_VsUHuwp-uPiG-22-242000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-608 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-528 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-626 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-665 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:14,078 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-704 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-766 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-863 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-831 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-765 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-893 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-792 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:16,346 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:scope2:Ifr-_VsUHuwp-uPiG-22-242000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:16,349 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:scope2:Ifr-_VsUHuwp-uPiG-22-242000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2533 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2624 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2680 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:16,723 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-959 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-964 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:17,282 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-57 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-129 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-30 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-181 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:19,131 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in default:scope1:3uLWrYql6tAJ-22-241000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-459 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-435 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-432 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-477 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:19,269 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:scope2:Ifr-_VsUHuwp-uPiG-22-242000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-36 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-14 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-51 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-148 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-140 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:19,651 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:scope1:R74D-22-239000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-717 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:20,697 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 00:48:20,782 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 173.919000149 sec 2021-05-03 00:48:21,319 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:scope1:R74D-22-239000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-129 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-30 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:21,394 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:scope1:R74D-22-239000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2596 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:21,394 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:scope1:R74D-22-239000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2596 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 00:48:24,960 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-03 00:48:24,960 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-03 00:48:24,961 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-03 00:48:24,961 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 00:48:24,969 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 00:48:24,969 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-03 00:48:24,970 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 00:48:24,971 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-03 00:48:35,641 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 00:50:21,746 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,750 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 3 docs from 0 to 500 2021-05-03 00:50:21,750 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,753 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,753 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,755 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,756 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,756 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,757 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,759 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 00:50:21,762 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 22 docs from 500 to 980 2021-05-03 00:50:21,765 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 21 docs from 500 to 980 2021-05-03 00:50:21,765 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 32 docs from 0 to 500 2021-05-03 00:50:21,766 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 00:50:21,766 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 35 docs from 500 to 980 2021-05-03 00:50:21,767 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 21 docs from 0 to 500 2021-05-03 00:50:21,769 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 00:50:21,772 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 26 docs from 500 to 980 2021-05-03 00:50:21,773 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 29 docs from 0 to 500 2021-05-03 00:50:21,773 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 5 docs from 2500 to 2900 2021-05-03 00:50:21,776 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 00:50:21,776 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-03 00:50:21,778 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-03 00:50:21,779 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 00:50:21,780 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 5 docs from 0 to 500 2021-05-03 00:50:21,782 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 00:50:21,783 | test | ERROR | pool-2-thread-15 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 00:55:12,381 | test | INFO | pool-2-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....*....X | 31.903 | | 21.100 | ....*X | 21.100 | | 5.099 | X | 5.099 | | 3.199 | **.....*....*.................**................................**.X | 219.687 | | 172.821 | ..............................................X | 172.821 | | 145.285 | ......................................X | 145.285 | | 141.285 | .....................................X | 141.285 | | 129.661 | ..................................X | 129.661 | | 106.825 | ............................*X | 106.825 | | 48.925 | ............*.....X | 66.554 | | 66.554 | .................*.........X | 101.554 | | 82.552 | .....................*.*X | 86.329 | | 77.929 | ....................*.....*......X | 117.600 | | 117.200 | ..............................*X | 117.200 | | 108.383 | ............................*..X | 115.583 | | 115.472 | ..............................**.....*....**.X | 151.002 | | 150.943 | ........................................X | 150.943 | | 148.926 | .......................................**..*.................***.....*............*..........**.......*X | 344.754 | | 339.093 | ...........................................................................................**........X | 367.022 | | 345.278 | .............................................................................................X | 345.278 | | 342.578 | ............................................................................................X | 342.578 | | 288.364 | .............................................................................**......*X | 311.100 | | 310.600 | ...................................................................................*..............*X | 359.800 | | 336.100 | ..........................................................................................X | 336.100 | | 267.700 | .......................................................................X | 267.700 | | 50.700 | ............*X | 50.700 | | 2.500 | *X | 2.500 | | 2.400 | X | 2.400 | | 0.000 | *************************************************.*.*X | 8.400 | | 7.400 | .X | 7.400 | | 6.300 | X | 6.300 | | 1.800 | *X | 1.800 | | 1.100 | *X | 1.100 | | 0.000 | ************X | 2.100 | | 2.000 | X | 2.000 | | 1.900 | X | 1.900 | | 1.800 | *******.*****.**X | 9.498 | | 7.498 | .X | 7.498 | | 3.500 | *X | 3.500 | | 1.700 | *X | 1.700 | | 0.800 | X | 0.800 | | 0.200 | *X | 0.200 | | 0.000 | **************X | 0.700 | | 0.500 | X | 0.500 | | 0.200 | *X | 0.200 | | 0.000 | ************.***X | 6.400 | | 5.300 | X | 5.300 | | 1.800 | *X | 1.800 | | 0.700 | X | 0.700 | | 0.000 | ******X | 1.600 | | 0.900 | *X | 0.900 | | 0.600 | X | 0.600 | | 0.600 | ***.**.X | 7.595 | | 5.196 | *X | 5.196 | | 2.901 | X | 2.901 | | 0.000 | *******X | 1.200 | | 0.400 | **X | 0.400 | | 0.000 | **********X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 00:55:12,605 | test | INFO | pool-2-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +----------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *****************..*..*....**.*X | 158.023 | | 111.223 | .....***.*X | 127.108 | | 118.861 | ......X | 118.861 | | 96.161 | ....*X | 96.161 | | 86.200 | ....X | 86.200 | | 54.300 | ..***...**...**...X | 198.207 | | 166.790 | .........***.*.X | 206.077 | | 164.149 | .........X | 164.149 | | 103.333 | .....X | 103.333 | | 89.898 | ....**.*..*X | 135.059 | | 99.418 | .....**.**..X | 152.343 | | 134.543 | .......X | 134.543 | | 115.451 | ......X | 115.451 | | 101.339 | .....***..****.......**.....**.......*................X | 705.900 | | 682.400 | .........................................**X | 689.600 | | 689.600 | .........................................X | 689.600 | | 603.300 | ....................................*..*X | 638.900 | | 608.300 | ....................................*..**X | 642.530 | | 635.630 | ......................................X | 635.630 | | 610.730 | ....................................*X | 610.730 | | 599.564 | ....................................**..*........**.....X | 843.529 | | 843.529 | ...................................................*......*...............*..*X | 1213.750 | | 1197.647 | .........................................................................X | 1197.647 | | 1147.613 | ......................................................................****.**X | 1173.698 | | 1140.398 | .....................................................................*..X | 1179.791 | | 1128.000 | ....................................................................**....*X | 1183.600 | | 1171.100 | .......................................................................*.*X | 1195.352 | | 1166.952 | .......................................................................*..*...........**....**.....*.....*.*X | 1616.292 | | 1582.092 | ................................................................................................*.***X | 1596.331 | | 1519.531 | .............................................................................................X | 1519.531 | | 1161.742 | ......................................................................*X | 1161.742 | | 1013.642 | .............................................................*X | 1013.642 | | 967.242 | ..........................................................**.....**..X | 1068.041 | | 1034.054 | ..............................................................X | 1034.054 | | 980.605 | ...........................................................*X | 980.605 | | 971.805 | ...........................................................***...**..*..............*.**X | 1304.171 | | 1301.168 | ...............................................................................**.*X | 1320.722 | | 1230.678 | ...........................................................................X | 1230.678 | | 1175.682 | .......................................................................*X | 1175.682 | | 1109.545 | ...................................................................X | 1109.545 | | 888.805 | .....................................................*X | 888.805 | | 855.423 | ...................................................*X | 855.423 | | 838.260 | ..................................................*....*.*X | 914.449 | | 888.478 | .....................................................X | 888.478 | | 738.429 | ............................................*X | 738.429 | | 660.129 | .......................................*X | 660.129 | | 596.700 | ...................................X | 596.700 | | 453.000 | ...........................*X | 453.000 | | 413.400 | ........................**....*.........*....*X | 682.437 | | 631.337 | ......................................X | 631.337 | | 504.692 | ..............................*X | 504.692 | | 493.507 | .............................**....................***..X | 845.954 | | 784.037 | ...............................................X | 784.037 | | 602.057 | ....................................*X | 602.057 | | 545.953 | ................................*X | 545.953 | | 507.950 | ..............................X | 507.950 | | 477.680 | ............................**.**..*.X | 541.400 | | 520.200 | ...............................*X | 520.200 | | 504.043 | ..............................X | 504.043 | | 435.643 | .........................**....**............................***.....*......*..................**......**.X | 1521.300 | | 1333.302 | .................................................................................X | 1333.302 | | 1141.045 | .....................................................................*X | 1141.045 | | 974.313 | ...........................................................X | 974.313 | | 691.429 | .........................................*X | 691.429 | | 684.740 | .........................................*X | 684.740 | | 552.180 | .................................**X | 552.180 | | 493.925 | .............................X | 493.925 | | 491.500 | .............................*..**....*X | 588.400 | | 508.800 | ..............................*X | 508.800 | | 457.300 | ...........................*X | 457.300 | | 440.562 | ..........................**X | 442.962 | | 385.662 | ......................*X | 385.662 | | 192.542 | ..........**.**.X | 210.281 | +----------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 00:55:13,917 | test | INFO | pool-2-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **......................................................................*....*.......X | 165.002 | | 114.802 | .......................................................X | 114.802 | | 111.600 | ......................................................*X | 111.600 | | 16.196 | .......*....**......X | 37.588 | | 28.593 | .............X | 28.593 | | 28.093 | ............**..........................*........*X | 96.291 | | 93.967 | .............................................*....X | 102.892 | | 89.392 | ...........................................*X | 89.392 | | 41.106 | ...................*X | 41.106 | | 36.206 | ................*...*.............***............**.....**...........***..*..**...X | 137.271 | | 134.500 | .................................................................X | 134.500 | | 120.887 | ..........................................................*X | 120.887 | | 113.307 | .......................................................*X | 114.501 | | 112.124 | ......................................................*..**...*X | 122.490 | | 118.871 | .........................................................**.......**.....*.X | 143.439 | | 112.324 | ......................................................X | 112.324 | | 103.965 | ..................................................**..*X | 108.374 | | 105.832 | ...................................................X | 105.832 | | 102.132 | .................................................*......**.**.........*........*....*X | 158.100 | | 143.400 | ......................................................................X | 143.400 | | 134.500 | .................................................................*X | 134.500 | | 131.200 | ................................................................**.........*...................*.......*X | 201.800 | | 200.300 | ..................................................................................................X | 200.300 | | 199.000 | .................................................................................................*X | 199.000 | | 189.384 | ............................................................................................**X | 189.384 | | 184.584 | ..........................................................................................**...X | 191.620 | | 175.820 | ......................................................................................X | 175.820 | | 124.610 | ............................................................*X | 124.610 | | 93.591 | .............................................*X | 93.591 | | 70.791 | ..................................X | 70.791 | | 8.200 | ...*X | 8.200 | | 0.000 | *******************************X | 1.200 | | 1.200 | X | 1.200 | | 0.800 | X | 0.800 | | 0.200 | *X | 0.200 | | 0.000 | *******************.*.***X | 5.801 | | 5.200 | .**X | 5.200 | | 4.800 | .**.***X | 7.898 | | 6.498 | ..X | 6.498 | | 3.800 | **.*......*.....................**............**.....*................*X | 127.000 | | 125.300 | .............................................................**X | 125.900 | | 119.270 | ..........................................................**X | 120.770 | | 119.570 | ..........................................................*.........**..**.X | 145.105 | | 133.173 | ................................................................*.*X | 133.314 | | 127.792 | ..............................................................X | 127.792 | | 120.212 | ..........................................................*X | 120.212 | | 111.091 | ......................................................*X | 111.091 | | 106.220 | ...................................................****X | 106.500 | | 104.100 | ..................................................****.X | 106.187 | | 99.187 | ................................................*X | 99.187 | | 91.580 | ............................................X | 91.580 | | 74.497 | ...................................X | 74.497 | | 35.897 | ................*X | 35.897 | | 25.705 | ...........*X | 25.705 | | 6.402 | ..*X | 6.402 | | 3.700 | **......*.........................*.............**....*..*X | 104.261 | | 92.378 | ............................................*X | 92.378 | | 87.553 | ..........................................X | 87.553 | | 75.239 | ....................................**X | 76.532 | | 76.400 | ....................................*........**..*X | 95.300 | | 84.900 | .........................................**X | 84.900 | | 82.000 | .......................................*..X | 86.567 | | 85.267 | .........................................*X | 85.267 | | 76.367 | ....................................*X | 76.367 | | 56.800 | ...........................X | 56.800 | | 8.000 | ..*X | 8.000 | | 0.000 | **X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 00:55:14,005 | test | WARNING | Thread-2396 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:-5E_pLP61DjO3a-22-62000 delete failed keys from task: [u'test_collections-0'] 2021-05-03 00:55:14,006 | test | WARNING | Thread-2401 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:EyRsc-22-51000 delete failed keys from task: [u'test_collections-0'] 2021-05-03 00:55:14,005 | test | WARNING | Thread-2400 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:in4-T_-22-67000 delete failed keys from task: [u'test_collections-0'] 2021-05-03 00:55:14,007 | test | WARNING | Thread-2404 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:rJzOR5jvsvl6aG-22-56000 delete failed keys from task: [u'test_collections-2'] 2021-05-03 00:55:14,007 | test | WARNING | Thread-2405 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:xsBthEWWh3dz-22-55000 delete failed keys from task: [u'test_collections-2'] 2021-05-03 00:55:14,009 | test | WARNING | Thread-2406 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:S-22-53000 delete failed keys from task: [u'test_collections-2'] 2021-05-03 00:55:14,010 | test | WARNING | Thread-2408 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:QfwimWuabd4_EUMT-22-62000 delete failed keys from task: [u'test_collections-2'] 2021-05-03 00:55:14,009 | test | WARNING | Thread-2407 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:LBQEG0_vOjmw-22-61000 delete failed keys from task: [u'test_collections-2'] 2021-05-03 00:55:14,058 | test | WARNING | Thread-2457 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:k-11-624000:vsL2O%20MEPIR9a-11-779000 create failed keys from task: [u'test_collections-801', u'test_collections-991', u'test_collections-1215', u'test_collections-1424', u'test_collections-1385', u'test_collections-1440'] 2021-05-03 00:55:14,145 | test | WARNING | Thread-2458 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:PRPR-11-370000 create failed keys from task: [u'test_collections-808', u'test_collections-827', u'test_collections-998', u'test_collections-1424', u'test_collections-1440', u'test_collections-1344'] 2021-05-03 00:55:14,145 | test | WARNING | Thread-2456 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:k-11-624000:0dE-11-780000 create failed keys from task: [u'test_collections-847', u'test_collections-802', u'test_collections-1424', u'test_collections-1440', u'test_collections-1069', u'test_collections-1380', u'test_collections-858', u'test_collections-972', u'test_collections-992', u'test_collections-1077', u'test_collections-1210'] 2021-05-03 00:55:14,160 | test | WARNING | Thread-2696 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:Wtj495VxWWqvls9hfxl-22-231000 delete failed keys from task: [u'test_collections-422', u'test_collections-474', u'test_collections-472'] 2021-05-03 00:55:14,161 | test | WARNING | Thread-2698 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:3uLWrYql6tAJ-22-241000 update failed keys from task: [u'test_collections-607', u'test_collections-527', u'test_collections-766', u'test_collections-522', u'test_collections-567', u'test_collections-765', u'test_collections-642', u'test_collections-611', u'test_collections-534', u'test_collections-697', u'test_collections-893', u'test_collections-571', u'test_collections-704', u'test_collections-702', u'test_collections-788', u'test_collections-863', u'test_collections-580', u'test_collections-519', u'test_collections-837', u'test_collections-711', u'test_collections-831', u'test_collections-754', u'test_collections-798', u'test_collections-872', u'test_collections-752', u'test_collections-792'] 2021-05-03 00:55:14,161 | test | WARNING | Thread-2699 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collections_2 update failed keys from task: [u'test_collections-729', u'test_collections-924', u'test_collections-769', u'test_collections-840', u'test_collections-961', u'test_collections-880', u'test_collections-937', u'test_collections-859', u'test_collections-815', u'test_collections-896', u'test_collections-850', u'test_collections-909', u'test_collections-827', u'test_collections-828', u'test_collections-946', u'test_collections-903', u'test_collections-701', u'test_collections-742', u'test_collections-862', u'test_collections-915', u'test_collections-757', u'test_collections-832'] 2021-05-03 00:55:14,161 | test | WARNING | Thread-2697 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 update failed keys from task: [u'test_collections-805', u'test_collections-727', u'test_collections-845', u'test_collections-964', u'test_collections-800', u'test_collections-921', u'test_collections-885', u'test_collections-938', u'test_collections-735', u'test_collections-856', u'test_collections-978', u'test_collections-899', u'test_collections-893', u'test_collections-770', u'test_collections-691', u'test_collections-629', u'test_collections-707', u'test_collections-868', u'test_collections-821', u'test_collections-864', u'test_collections-620', u'test_collections-780', u'test_collections-781', u'test_collections-959', u'test_collections-718', u'test_collections-837', u'test_collections-636', u'test_collections-878', u'test_collections-797', u'test_collections-633', u'test_collections-831', u'test_collections-754', u'test_collections-872', u'test_collections-752', u'test_collections-791'] 2021-05-03 00:55:14,233 | test | WARNING | Thread-2459 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:kLN2qqzzMmGlDuCSakfI-11-191000:Nq9JsGIGdBjT-11-371000 create failed keys from task: [u'test_collections-808', u'test_collections-827', u'test_collections-1380', u'test_collections-998', u'test_collections-1424', u'test_collections-1077', u'test_collections-1440', u'test_collections-1210', u'test_collections-1069'] 2021-05-03 00:55:14,250 | test | WARNING | Thread-2883 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:wN-E9T_7F-12-438000:e_JdCR-12-647000 create failed keys from task: [u'test_collections-525', u'test_collections-721', u'test_collections-487', u'test_collections-1027', u'test_collections-1161', u'test_collections-1187', u'test_collections-517', u'test_collections-457', u'test_collections-556', u'test_collections-870', u'test_collections-1017'] 2021-05-03 00:55:14,250 | test | WARNING | Thread-2882 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:wN-E9T_7F-12-438000:Rx-12-680000 create failed keys from task: [u'test_collections-525', u'test_collections-721', u'test_collections-487', u'test_collections-1027', u'test_collections-1161', u'test_collections-1187', u'test_collections-517', u'test_collections-556', u'test_collections-457', u'test_collections-870', u'test_collections-1017'] 2021-05-03 00:55:14,256 | test | WARNING | Thread-2804 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aOi%-11-981000:j7WFG%yH8q0svVd-12-134000 create failed keys from task: [u'test_collections-566'] 2021-05-03 00:55:14,286 | test | WARNING | Thread-2711 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:R74D-22-239000 update failed keys from task: [u'test_collections-608', u'test_collections-867', u'test_collections-623', u'test_collections-602', u'test_collections-685', u'test_collections-681', u'test_collections-660', u'test_collections-618', u'test_collections-717', u'test_collections-513', u'test_collections-692'] 2021-05-03 00:55:14,289 | test | WARNING | Thread-2700 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:qU-22-233000 update failed keys from task: [u'test_collections-827', u'test_collections-729', u'test_collections-828', u'test_collections-707', u'test_collections-946', u'test_collections-968', u'test_collections-903', u'test_collections-845', u'test_collections-769', u'test_collections-868', u'test_collections-789', u'test_collections-742', u'test_collections-784', u'test_collections-840', u'test_collections-915', u'test_collections-959', u'test_collections-815', u'test_collections-757', u'test_collections-797', u'test_collections-850', u'test_collections-691'] 2021-05-03 00:55:14,323 | test | WARNING | Thread-2501 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:hLVhBK3qIm-10-951000 create failed keys from task: [u'test_collections-1312', u'test_collections-1632', u'test_collections-1524', u'test_collections-1647', u'test_collections-1282', u'test_collections-1065'] 2021-05-03 00:55:14,328 | test | WARNING | Thread-2731 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:Ifr-_VsUHuwp-uPiG-22-242000 update failed keys from task: [u'test_collections-618', u'test_collections-608', u'test_collections-528', u'test_collections-626', u'test_collections-549', u'test_collections-568', u'test_collections-665', u'test_collections-692'] 2021-05-03 00:55:14,365 | test | WARNING | Thread-2699 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collections_2 delete failed keys from task: [u'test_collections-12', u'test_collections-57', u'test_collections-129', u'test_collections-30', u'test_collections-181'] 2021-05-03 00:55:14,368 | test | WARNING | Thread-2500 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Q-HqMllcWETkCTHa0uvU-10-795000:B0jhs0rgkB8XDcdwgt-10-952000 create failed keys from task: [u'test_collections-1312', u'test_collections-1257', u'test_collections-1282', u'test_collections-1065', u'test_collections-1562'] 2021-05-03 00:55:14,368 | test | WARNING | Thread-2503 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:scope1:6Hoo3BtK2Jk1aUs-22-71000 delete failed keys from task: [u'test_collections-2'] 2021-05-03 00:55:14,418 | test | WARNING | Thread-2711 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:R74D-22-239000 delete failed keys from task: [u'test_collections-129', u'test_collections-30'] 2021-05-03 00:55:14,434 | test | WARNING | Thread-2711 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:R74D-22-239000 create failed keys from task: [u'test_collections-2596'] 2021-05-03 00:55:14,440 | test | WARNING | Thread-2731 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:Ifr-_VsUHuwp-uPiG-22-242000 delete failed keys from task: [u'test_collections-36', u'test_collections-14', u'test_collections-35', u'test_collections-73', u'test_collections-51', u'test_collections-148', u'test_collections-187', u'test_collections-140'] 2021-05-03 00:55:14,463 | test | WARNING | Thread-2698 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:3uLWrYql6tAJ-22-241000 delete failed keys from task: [u'test_collections-283', u'test_collections-163', u'test_collections-280', u'test_collections-44', u'test_collections-459', u'test_collections-175', u'test_collections-130', u'test_collections-293', u'test_collections-291', u'test_collections-37', u'test_collections-76', u'test_collections-72', u'test_collections-346', u'test_collections-303', u'test_collections-224', u'test_collections-301', u'test_collections-266', u'test_collections-186', u'test_collections-261', u'test_collections-2', u'test_collections-69', u'test_collections-118', u'test_collections-313', u'test_collections-435', u'test_collections-355', u'test_collections-432', u'test_collections-477', u'test_collections-353', u'test_collections-310'] 2021-05-03 00:55:14,549 | test | WARNING | Thread-2731 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:Ifr-_VsUHuwp-uPiG-22-242000 create failed keys from task: [u'test_collections-2533', u'test_collections-2624', u'test_collections-2680'] 2021-05-03 00:55:14,559 | test | WARNING | Thread-2700 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:qU-22-233000 delete failed keys from task: [u'test_collections-207', u'test_collections-327', u'test_collections-324', u'test_collections-203', u'test_collections-97', u'test_collections-322', u'test_collections-169', u'test_collections-246', u'test_collections-362', u'test_collections-218', u'test_collections-139', u'test_collections-42', u'test_collections-215', u'test_collections-254', u'test_collections-211', u'test_collections-255', u'test_collections-497', u'test_collections-296', u'test_collections-330', u'test_collections-250', u'test_collections-33', u'test_collections-306', u'test_collections-388', u'test_collections-345', u'test_collections-461', u'test_collections-385', u'test_collections-381', u'test_collections-359', u'test_collections-434', u'test_collections-397', u'test_collections-393', u'test_collections-471'] 2021-05-03 00:55:14,676 | test | WARNING | Thread-2697 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 delete failed keys from task: [u'test_collections-18', u'test_collections-58', u'test_collections-33', u'test_collections-327', u'test_collections-203', u'test_collections-97', u'test_collections-289', u'test_collections-169', u'test_collections-246', u'test_collections-144', u'test_collections-362', u'test_collections-461', u'test_collections-141', u'test_collections-319', u'test_collections-139', u'test_collections-434', u'test_collections-42', u'test_collections-255', u'test_collections-393', u'test_collections-152', u'test_collections-471'] 2021-05-03 00:55:14,700 | test | WARNING | Thread-2698 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:3uLWrYql6tAJ-22-241000 create failed keys from task: [u'test_collections-2537', u'test_collections-2592', u'test_collections-2882', u'test_collections-2597', u'test_collections-2663'] 2021-05-03 00:55:14,723 | test | WARNING | Thread-2697 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:MADtkoBTCIvbXPTBtTv-22-235000 create failed keys from task: [u'test_collections-2838', u'test_collections-2759'] 2021-05-03 00:55:14,726 | test | WARNING | Thread-2700 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:qU-22-233000 create failed keys from task: [u'test_collections-2838', u'test_collections-2759'] 2021-05-03 00:55:28,496 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 00:57:16,019 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 00:57:16,019 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 00:57:39,546 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 00:57:39,546 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 96% 2021-05-03 00:57:52,311 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 00:57:52,312 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 00:58:35,427 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 00:59:19,864 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1679999828 seconds to complete 2021-05-03 00:59:19,887 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2530.420s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_1 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_2', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 2, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 00:59:30,221 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 00:59:31,256 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 00:59:31,983 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d53d type=client open_count=1 channel=[id: 0xb7a9ddeb, 0.0.0.0/0.0.0.0:53593] timeout=300.0>) 2021-05-03 00:59:37,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d53e type=client open_count=1 channel=[id: 0xc15ca9ea, 0.0.0.0/0.0.0.0:49173] timeout=300.0>) 2021-05-03 00:59:39,509 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d53f type=client open_count=1 channel=[id: 0xf0d1a623, 0.0.0.0/0.0.0.0:57691] timeout=300.0>) 2021-05-03 00:59:41,538 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d540 type=client open_count=1 channel=[id: 0xbc6bd88f, 0.0.0.0/0.0.0.0:33143] timeout=300.0>) 2021-05-03 00:59:43,565 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d541 type=client open_count=1 channel=[id: 0xdcab2bd6, 0.0.0.0/0.0.0.0:54893] timeout=300.0>) 2021-05-03 00:59:45,605 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d542 type=client open_count=1 channel=[id: 0x876f8df9, 0.0.0.0/0.0.0.0:44865] timeout=300.0>) 2021-05-03 00:59:47,651 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d543 type=client open_count=1 channel=[id: 0xb63f67d6, 0.0.0.0/0.0.0.0:59017] timeout=300.0>) 2021-05-03 00:59:49,697 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d544 type=client open_count=1 channel=[id: 0x834582b3, 0.0.0.0/0.0.0.0:38904] timeout=300.0>) 2021-05-03 00:59:51,740 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d545 type=client open_count=1 channel=[id: 0x0db7e671, 0.0.0.0/0.0.0.0:48199] timeout=300.0>) 2021-05-03 00:59:53,765 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d546 type=client open_count=1 channel=[id: 0xf9ed6031, 0.0.0.0/0.0.0.0:54221] timeout=300.0>) 2021-05-03 01:00:01,730 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d547 type=client open_count=1 channel=[id: 0x1708016c, 0.0.0.0/0.0.0.0:34866] timeout=300.0>) 2021-05-03 01:00:03,755 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d548 type=client open_count=1 channel=[id: 0xe0d6de8e, 0.0.0.0/0.0.0.0:60407] timeout=300.0>) 2021-05-03 01:00:05,786 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d549 type=client open_count=1 channel=[id: 0x086ea61b, 0.0.0.0/0.0.0.0:34643] timeout=300.0>) 2021-05-03 01:00:07,848 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d54a type=client open_count=1 channel=[id: 0x714fb761, 0.0.0.0/0.0.0.0:42997] timeout=300.0>) 2021-05-03 01:00:09,872 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d54b type=client open_count=1 channel=[id: 0xb3ed75fb, 0.0.0.0/0.0.0.0:60763] timeout=300.0>) 2021-05-03 01:00:11,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d54c type=client open_count=1 channel=[id: 0xee3e75cb, 0.0.0.0/0.0.0.0:32793] timeout=300.0>) 2021-05-03 01:00:13,917 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d54d type=client open_count=1 channel=[id: 0x01425436, 0.0.0.0/0.0.0.0:54699] timeout=300.0>) 2021-05-03 01:00:15,941 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d54e type=client open_count=1 channel=[id: 0x95f054b8, 0.0.0.0/0.0.0.0:54355] timeout=300.0>) 2021-05-03 01:00:17,964 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d54f type=client open_count=1 channel=[id: 0x810d592b, 0.0.0.0/0.0.0.0:38105] timeout=300.0>) 2021-05-03 01:00:25,763 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d550 type=client open_count=1 channel=[id: 0xfadd8494, 0.0.0.0/0.0.0.0:38720] timeout=300.0>) 2021-05-03 01:00:27,789 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d551 type=client open_count=1 channel=[id: 0x4467b5c6, 0.0.0.0/0.0.0.0:53181] timeout=300.0>) 2021-05-03 01:00:29,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d552 type=client open_count=1 channel=[id: 0x8433848f, 0.0.0.0/0.0.0.0:41001] timeout=300.0>) 2021-05-03 01:00:31,842 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d553 type=client open_count=1 channel=[id: 0xe23d0282, 0.0.0.0/0.0.0.0:40245] timeout=300.0>) 2021-05-03 01:00:33,868 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d554 type=client open_count=1 channel=[id: 0xf0af415c, 0.0.0.0/0.0.0.0:42315] timeout=300.0>) 2021-05-03 01:00:35,903 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d555 type=client open_count=1 channel=[id: 0x58b1aea4, 0.0.0.0/0.0.0.0:57765] timeout=300.0>) 2021-05-03 01:00:37,927 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d556 type=client open_count=1 channel=[id: 0xc422a0b6, 0.0.0.0/0.0.0.0:57971] timeout=300.0>) 2021-05-03 01:00:39,954 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d557 type=client open_count=1 channel=[id: 0xd869becb, 0.0.0.0/0.0.0.0:37369] timeout=300.0>) 2021-05-03 01:00:41,976 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d558 type=client open_count=1 channel=[id: 0x25811cce, 0.0.0.0/0.0.0.0:54765] timeout=300.0>) 2021-05-03 01:00:44,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d559 type=client open_count=1 channel=[id: 0xc8954185, 0.0.0.0/0.0.0.0:53838] timeout=300.0>) 2021-05-03 01:00:52,233 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d55a type=client open_count=1 channel=[id: 0x3707c5b7, 0.0.0.0/0.0.0.0:41245] timeout=300.0>) 2021-05-03 01:00:54,256 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d55b type=client open_count=1 channel=[id: 0xaf245b39, 0.0.0.0/0.0.0.0:53255] timeout=300.0>) 2021-05-03 01:00:56,278 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d55c type=client open_count=1 channel=[id: 0x606c652e, 0.0.0.0/0.0.0.0:42416] timeout=300.0>) 2021-05-03 01:00:58,303 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d55d type=client open_count=1 channel=[id: 0x8e1e207e, 0.0.0.0/0.0.0.0:52072] timeout=300.0>) 2021-05-03 01:01:00,326 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d55e type=client open_count=1 channel=[id: 0x3faa482d, 0.0.0.0/0.0.0.0:57194] timeout=300.0>) 2021-05-03 01:01:02,349 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d55f type=client open_count=1 channel=[id: 0x1c8be456, 0.0.0.0/0.0.0.0:32839] timeout=300.0>) 2021-05-03 01:01:04,371 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d560 type=client open_count=1 channel=[id: 0xeaee08d5, 0.0.0.0/0.0.0.0:51416] timeout=300.0>) 2021-05-03 01:01:06,395 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d561 type=client open_count=1 channel=[id: 0x6505a3e6, 0.0.0.0/0.0.0.0:59421] timeout=300.0>) 2021-05-03 01:01:08,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d562 type=client open_count=1 channel=[id: 0xfebdf925, 0.0.0.0/0.0.0.0:43043] timeout=300.0>) 2021-05-03 01:01:13,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d563 type=client open_count=1 channel=[id: 0xa4fdc487, 0.0.0.0/0.0.0.0:40259] timeout=300.0>) 2021-05-03 01:01:15,644 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d564 type=client open_count=1 channel=[id: 0x1e43e15f, 0.0.0.0/0.0.0.0:60553] timeout=300.0>) 2021-05-03 01:01:17,677 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d565 type=client open_count=1 channel=[id: 0x97dc55e9, 0.0.0.0/0.0.0.0:42941] timeout=300.0>) 2021-05-03 01:01:19,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d566 type=client open_count=1 channel=[id: 0x4ae65034, 0.0.0.0/0.0.0.0:53792] timeout=300.0>) 2021-05-03 01:01:21,726 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d567 type=client open_count=1 channel=[id: 0xa7edea51, 0.0.0.0/0.0.0.0:40103] timeout=300.0>) 2021-05-03 01:01:23,752 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 01:01:40,407 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 01:01:54,694 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 01:01:54,720 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 01:01:54,766 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 01:02:34,232 | test | INFO | pool-4-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 14.4444444444 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 01:02:34,285 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 01:02:44,450 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 01:02:44,492 | test | INFO | pool-4-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2599999905 sec 2021-05-03 01:02:44,849 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 01:03:22,815 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 01:03:23,338 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 538142808 | 175467855 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 509652488 | 137671199 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 594619072 | 131096731 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 01:03:23,339 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 01:04:26,355 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 01:04:26,372 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 01:16:05,141 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *....*.............**......*X | 13.501 | | 10.800 | .................X | 10.800 | | 3.600 | .....X | 3.600 | | 0.000 | ***************.*...***X | 2.600 | | 2.000 | ..X | 2.000 | | 0.400 | *X | 0.400 | | 0.000 | ******************************.**.X | 1.500 | | 1.200 | .X | 1.200 | | 0.400 | *X | 0.400 | | 0.000 | *****************.**.***.*..**..*....**.**.............*.................................................*..........................*X | 58.195 | | 53.095 | ..........................................................................................X | 53.095 | | 22.500 | .....................................X | 22.500 | | 7.700 | ............*X | 7.700 | | 4.100 | ......X | 4.100 | | 3.800 | .....**.**..*.......X | 9.700 | | 8.500 | .............*X | 8.500 | | 6.400 | .........X | 6.400 | | 0.800 | **.**..*.....*.***..***X | 7.502 | | 6.902 | ..........*.............*....**....*X | 18.899 | | 7.700 | ............X | 7.700 | | 3.300 | ....*X | 3.300 | | 0.900 | *.................**........*X | 15.300 | | 7.500 | ...........**X | 7.500 | | 3.499 | .....X | 3.499 | | 1.399 | .*...*X | 2.999 | | 2.200 | ..**.*.*..*X | 4.500 | | 3.900 | .....**.**......**...........*....*X | 16.701 | | 12.801 | ....................X | 12.801 | | 4.002 | .....*X | 4.002 | | 1.600 | .**..*......X | 6.000 | | 4.400 | ......*X | 4.400 | | 3.200 | ....X | 3.200 | | 0.000 | *****.*..**......*.............*........*X | 17.699 | | 15.899 | ..........................*X | 15.899 | | 11.901 | ...................X | 11.901 | | 7.899 | ............**X | 7.899 | | 6.298 | .........*X | 6.298 | | 5.100 | .......*X | 5.100 | | 3.900 | .....X | 3.900 | | 1.500 | .X | 1.500 | | 1.500 | .*X | 1.500 | | 1.200 | .X | 1.200 | | 0.400 | *X | 0.400 | | 0.000 | ******.*.X | 1.500 | | 1.100 | *X | 1.100 | | 0.800 | X | 0.800 | | 0.000 | **...**.........*X | 7.500 | | 5.500 | ........*X | 5.500 | | 4.300 | ......*.....**..........*.......*X | 17.000 | | 14.599 | ........................X | 14.599 | | 9.399 | ...............*X | 9.399 | | 5.199 | .......X | 5.199 | | 0.900 | ***.X | 1.200 | | 1.000 | *****..X | 2.300 | | 2.300 | ..**.***.*X | 3.300 | | 2.700 | ...X | 2.700 | | 2.699 | ...*.**.**X | 3.701 | | 3.300 | ....*X | 3.300 | | 2.700 | ...X | 2.700 | | 1.900 | ..*X | 1.900 | | 1.100 | ****..**.*...X | 4.500 | | 4.499 | ......**.*.*X | 5.599 | | 5.200 | .......*X | 5.200 | | 4.600 | ......X | 4.600 | | 3.001 | ....*X | 3.001 | | 3.000 | ....X | 3.000 | | 2.400 | ...X | 2.400 | | 0.800 | *X | 0.800 | | 0.000 | *********..**...*.......*X | 7.401 | | 7.399 | ...........X | 7.399 | | 5.599 | ........*X | 5.599 | | 1.199 | .X | 1.199 | | 0.000 | ************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 01:16:06,200 | test | INFO | pool-4-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...**..............................*.................**.....................*............................*.*X | 1366.425 | | 1254.096 | ..........................................................................................X | 1254.096 | | 1147.896 | ...................................................................................*X | 1147.896 | | 1112.035 | ................................................................................*X | 1119.397 | | 1064.297 | ............................................................................**.*X | 1066.833 | | 1055.395 | ............................................................................**X | 1057.395 | | 1044.495 | ...........................................................................X | 1044.495 | | 1024.595 | .........................................................................X | 1024.595 | | 944.095 | ....................................................................*X | 944.095 | | 888.495 | ................................................................*X | 888.495 | | 837.900 | ............................................................X | 837.900 | | 675.400 | ................................................*X | 675.400 | | 675.100 | ................................................X | 675.100 | | 671.800 | ................................................*..*X | 702.273 | | 686.573 | .................................................**.*..*...***....**.*X | 833.751 | | 833.551 | ............................................................X | 833.551 | | 821.200 | ...........................................................*X | 821.200 | | 797.200 | .........................................................**.X | 808.400 | | 770.500 | .......................................................*X | 770.500 | | 757.900 | ......................................................X | 757.900 | | 708.500 | ..................................................X | 708.500 | | 614.100 | ...........................................*X | 614.100 | | 603.400 | ...........................................*X | 603.400 | | 601.600 | ...........................................*..****..*....*...**.*..**.*X | 806.803 | | 789.821 | ........................................................*X | 792.510 | | 789.853 | ........................................................***...X | 820.611 | | 766.296 | .......................................................X | 766.296 | | 744.128 | .....................................................*X | 744.128 | | 683.864 | .................................................X | 683.864 | | 627.465 | ............................................*X | 627.465 | | 598.768 | ..........................................*X | 598.768 | | 587.175 | .........................................X | 587.175 | | 533.721 | ......................................***...*........**...*X | 731.400 | | 720.100 | ...................................................X | 720.100 | | 667.249 | ...............................................X | 667.249 | | 629.649 | .............................................***.....**...*..*.*X | 785.468 | | 762.727 | ......................................................*X | 762.727 | | 715.927 | ...................................................X | 715.927 | | 583.927 | .........................................X | 583.927 | | 561.600 | ........................................*X | 561.600 | | 554.444 | .......................................*....****..*...*..**..*..*.*X | 768.754 | | 742.254 | .....................................................*X | 742.254 | | 683.276 | .................................................X | 683.276 | | 676.776 | ................................................**...*.......X | 808.032 | | 794.432 | .........................................................*X | 794.432 | | 757.032 | ......................................................X | 757.032 | | 669.100 | ...............................................*X | 669.100 | | 668.800 | ...............................................*X | 668.800 | | 653.000 | ..............................................X | 653.000 | | 585.100 | .........................................*X | 585.100 | | 571.005 | ........................................*..*.......**.**.X | 711.020 | | 705.720 | ..................................................*.*X | 716.600 | | 705.700 | ..................................................X | 705.700 | | 701.600 | ..................................................X | 701.600 | | 695.729 | .................................................**..*.*X | 732.801 | | 715.035 | ...................................................*X | 715.035 | | 669.735 | ................................................X | 669.735 | | 570.539 | ........................................X | 570.539 | | 570.039 | ........................................**.**.*.**..*......**.X | 730.104 | | 722.504 | ...................................................X | 722.504 | | 712.306 | ...................................................***X | 722.348 | | 698.353 | ..................................................*X | 698.353 | | 696.980 | ..................................................**.*..X | 745.437 | | 738.085 | .....................................................*X | 738.085 | | 718.685 | ...................................................X | 718.685 | | 647.548 | ..............................................*X | 647.548 | | 637.941 | .............................................*X | 637.941 | | 569.697 | ........................................**X | 569.697 | | 556.222 | .......................................*.*.....**.**...*........*...**.X | 849.700 | | 798.500 | .........................................................X | 798.500 | | 767.800 | .......................................................**.*.......**.*X | 891.400 | | 643.300 | ..............................................**X | 643.300 | | 582.300 | .........................................X | 582.300 | | 554.600 | .......................................X | 554.600 | | 525.700 | .....................................****...**.**.*.....**....**.*......X | 808.054 | | 802.454 | .........................................................**..*.*X | 836.850 | | 824.220 | ...........................................................*X | 824.220 | | 808.768 | ..........................................................X | 808.768 | | 731.055 | ....................................................X | 731.055 | | 706.411 | ..................................................*X | 706.411 | | 640.411 | .............................................X | 640.411 | | 521.406 | .....................................*X | 521.406 | | 514.673 | ....................................**...*.....**..**.****..*......**.*X | 780.300 | | 741.100 | .....................................................****.X | 751.981 | | 700.648 | ..................................................*X | 700.648 | | 686.391 | .................................................*X | 686.391 | | 582.121 | .........................................**X | 582.121 | | 556.653 | .......................................*.*.*X | 574.500 | | 571.400 | ........................................*X | 571.400 | | 564.400 | ........................................***.*..*...**..**..*...X | 747.076 | | 740.030 | .....................................................*X | 740.030 | | 732.930 | ....................................................X | 732.930 | | 728.030 | ....................................................**.**X | 744.776 | | 658.276 | ...............................................X | 658.276 | | 634.349 | .............................................*X | 634.349 | | 598.849 | ..........................................*....*X | 644.149 | | 628.945 | .............................................*X | 628.945 | | 610.545 | ...........................................X | 610.545 | | 480.295 | ..................................X | 480.295 | +----------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 01:16:08,121 | test | INFO | pool-4-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *....................*.....................................................**...........................*X | 60.002 | | 48.000 | ..............................................................................X | 48.000 | | 16.000 | .........................X | 16.000 | | 0.000 | ************.**.**.*X | 2.200 | | 1.400 | .**.*....**...*X | 6.400 | | 5.500 | ........X | 5.500 | | 3.100 | ....*X | 3.100 | | 1.100 | X | 1.100 | | 0.800 | X | 0.800 | | 0.000 | ***********************************************************.....*.............................**...............**X | 29.697 | | 14.300 | ......................*X | 14.300 | | 6.300 | .........X | 6.300 | | 2.700 | ...X | 2.700 | | 0.800 | **..**..................**X | 13.000 | | 13.000 | ....................X | 13.000 | | 10.000 | ...............X | 10.000 | | 2.000 | ..*X | 2.000 | | 1.200 | **................**..*X | 11.698 | | 10.497 | ................X | 10.497 | | 4.100 | .....****..*....................*....**.X | 19.899 | | 7.100 | ..........X | 7.100 | | 2.700 | ...*X | 2.700 | | 1.500 | .*........**....*X | 8.700 | | 7.500 | ...........X | 7.500 | | 2.700 | ...X | 2.700 | | 1.897 | ..**.*.....*X | 5.997 | | 4.801 | .......**...................***.........X | 21.800 | | 19.400 | ...............................X | 19.400 | | 11.400 | .................*X | 11.400 | | 7.799 | ...........*X | 7.799 | | 7.299 | ...........X | 7.299 | | 7.299 | ...........***X | 7.701 | | 6.901 | ..........X | 6.901 | | 3.701 | .....*X | 3.701 | | 1.700 | .X | 1.700 | | 1.500 | .**X | 1.500 | | 1.100 | *X | 1.100 | | 0.000 | *********..*.....*..*X | 5.600 | | 4.400 | ......X | 4.400 | | 1.200 | X | 1.200 | | 0.400 | **.*.**.**.*....X | 5.201 | | 5.200 | .......*X | 5.200 | | 4.000 | .....*X | 4.000 | | 0.800 | X | 0.800 | | 0.400 | **.*.*..**..**.......X | 8.200 | | 7.400 | ...........*X | 7.400 | | 5.600 | ........X | 5.600 | | 3.200 | ....*......**.....**.......X | 14.100 | | 10.100 | ...............*X | 10.100 | | 7.100 | ..........X | 7.100 | | 2.300 | ..***.*X | 2.600 | | 1.800 | .**X | 1.800 | | 1.200 | X | 1.200 | | 0.400 | X | 0.400 | | 0.000 | ***********.*....**..*X | 4.500 | | 2.299 | ..*X | 2.299 | | 1.500 | .**.*...**..**.*.X | 6.400 | | 5.600 | ........*X | 5.600 | | 5.300 | .......X | 5.300 | | 4.500 | ......**.***...X | 6.900 | | 6.499 | .........**.X | 6.799 | | 4.699 | ......*X | 4.699 | | 3.900 | .....*X | 3.900 | | 3.300 | ....X | 3.300 | | 2.701 | ...X | 2.701 | | 1.900 | ..*X | 1.900 | | 1.300 | .X | 1.300 | | 0.000 | *************.*....**..X | 4.499 | | 3.599 | ....*X | 3.599 | | 1.199 | X | 1.199 | | 0.000 | *************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 01:16:09,348 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 45.2119309262 | 3693387776 | 2628780032 | 0 / 3909087232 | 101855 / 204706 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 37.0148476166 | 3693387776 | 2683961344 | 0 / 3909087232 | 101600 / 202400 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 44.996121024 | 6078930944 | 4841861120 | 0 / 3758092288 | 101722 / 197821 | 7.0.0-5085-enterprise | | 172.23.120.206 | kv | 15.6090071648 | 3691520000 | 2778419200 | 0 / 0 | 99147 / 202853 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 48.8221589438 | 4201836544 | 3153895424 | 0 / 3758092288 | 101301 / 203455 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 01:16:24,076 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 01:18:19,904 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 157814024 | 248975984 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 157812216 | 279038274 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 435078296 | 443342281 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-03 01:18:19,905 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 01:18:30,194 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_out 2021-05-03 01:18:30,194 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_out 2021-05-03 01:18:30,313 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 01:18:30,605 | test | INFO | pool-4-thread-9 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 13.8300586884 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 17.5890826384 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 8.86974151039 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 6.0445118867 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.38083034341 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 01:18:30,671 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 01:18:37,390 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 01:18:40,775 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 01:18:45,913 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 4.31 2021-05-03 01:18:51,078 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 7.31 2021-05-03 01:18:56,203 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 10.26 2021-05-03 01:19:01,338 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 13.01 2021-05-03 01:19:06,505 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 15.76 2021-05-03 01:19:11,624 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 18.11 2021-05-03 01:19:16,752 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 20.73 2021-05-03 01:19:21,880 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 23.26 2021-05-03 01:19:27,302 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 25.88 2021-05-03 01:19:32,555 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 28.68 2021-05-03 01:19:37,697 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 31.36 2021-05-03 01:19:42,937 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 01:19:48,078 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 01:19:55,720 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.38 2021-05-03 01:20:01,384 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 38.56 2021-05-03 01:20:02,380 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Kmw1IEJXqudQA-34-976000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-30 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-20 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-157 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:02,381 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Kmw1IEJXqudQA-34-976000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-30 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-20 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-157 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:06,483 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 42.18 2021-05-03 01:20:11,536 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-59 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:11,538 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-59 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:11,641 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 44.46 2021-05-03 01:20:13,865 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Kmw1IEJXqudQA-34-976000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-532 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:13,867 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Kmw1IEJXqudQA-34-976000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-532 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:16,756 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 47.07 2021-05-03 01:20:21,828 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Kmw1IEJXqudQA-34-976000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-733 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:21,829 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Kmw1IEJXqudQA-34-976000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-733 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:21,967 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 49.35 2021-05-03 01:20:24,216 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:24,217 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-408 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-448 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-598 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:27,456 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 51.85 2021-05-03 01:20:28,101 | test | INFO | pool-4-thread-1 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:fSvY0hSTCqB0_h5DUD-34-458000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:28,104 | test | INFO | pool-4-thread-1 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:fSvY0hSTCqB0_h5DUD-34-458000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-143 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:29,523 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-85 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:29,525 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-85 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:29,526 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-19 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-92 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:29,526 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-19 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-85 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-92 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:32,701 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 53.85 2021-05-03 01:20:37,997 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 56.01 2021-05-03 01:20:41,730 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:41,732 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:41,765 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:41,766 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:43,247 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 58.12 2021-05-03 01:20:47,801 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:47,802 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-923 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:48,555 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 60.49 2021-05-03 01:20:53,740 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 63.02 2021-05-03 01:20:54,677 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:54,677 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:54,678 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:54,678 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:20:58,951 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.36 2021-05-03 01:21:02,111 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:02,111 | test | INFO | pool-4-thread-18 | [table_view:display:72] Keys failed in bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1064 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:04,229 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 01:21:06,187 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:06,187 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:hcchR-34-251000:WPAWrA0J-34-457000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-761 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:06,209 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:06,209 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in bucket2:KY8-35-122000:En7vNv5jc-35-226000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-761 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:09,604 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 01:21:16,368 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 67.88 2021-05-03 01:21:17,329 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:3c3l6Bro-36-991000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3725 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:17,331 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:3c3l6Bro-36-991000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3725 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:21,694 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 72.9 2021-05-03 01:21:26,213 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:3c3l6Bro-36-991000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4080 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:26,214 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:3c3l6Bro-36-991000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4080 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4110 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:26,229 | test | INFO | pool-4-thread-11 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:ETs0-36-992000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4080 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:26,230 | test | INFO | pool-4-thread-11 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:ETs0-36-992000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4080 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4110 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:27,029 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 76.47 2021-05-03 01:21:28,677 | test | INFO | pool-4-thread-5 | [table_view:display:72] Keys failed in default:5QOcV1hUe-36-568000:5NrUP-36-691000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4028 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:28,678 | test | INFO | pool-4-thread-5 | [table_view:display:72] Keys failed in default:5QOcV1hUe-36-568000:5NrUP-36-691000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4086 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4116 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4028 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:30,346 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1816 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1986 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:30,348 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1986 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:31,572 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:0%pF3gCpSLoIme4-36-413000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1714 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1684 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:31,572 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:0%pF3gCpSLoIme4-36-413000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1757 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1714 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1684 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:31,592 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in default:5QOcV1hUe-36-568000:pb-36-692000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:31,595 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in default:5QOcV1hUe-36-568000:pb-36-692000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4605 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4795 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:32,220 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 78.74 2021-05-03 01:21:37,466 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 81.25 2021-05-03 01:21:42,766 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 82.99 2021-05-03 01:21:42,881 | test | INFO | pool-4-thread-11 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:ETs0-36-992000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4474 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:42,882 | test | INFO | pool-4-thread-11 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:ETs0-36-992000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4474 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:42,887 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:3c3l6Bro-36-991000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4474 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:42,888 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:DCdXwh-36-882000:3c3l6Bro-36-991000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4474 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:44,865 | test | INFO | pool-4-thread-5 | [table_view:display:72] Keys failed in default:5QOcV1hUe-36-568000:5NrUP-36-691000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:44,865 | test | INFO | pool-4-thread-5 | [table_view:display:72] Keys failed in default:5QOcV1hUe-36-568000:5NrUP-36-691000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4408 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4598 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:47,450 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2343 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:47,451 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2343 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2223 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:48,138 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 85.31 2021-05-03 01:21:53,368 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 87.16 2021-05-03 01:21:53,891 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2470 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:53,891 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2491 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2501 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2470 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:58,596 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 89.21 2021-05-03 01:21:59,239 | test | INFO | pool-4-thread-14 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:uylFiogRHveaj-45-27000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-102 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-140 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:59,243 | test | INFO | pool-4-thread-26 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:uylFiogRHveaj-45-27000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-632 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-510 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-670 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:21:59,246 | test | INFO | pool-4-thread-25 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-632 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-565 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-510 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-562 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-670 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:00,756 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-102 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-140 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:03,075 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 3 docs from 0 to 5000 2021-05-03 01:22:03,076 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 4 docs from 0 to 5000 2021-05-03 01:22:03,078 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 01:22:03,078 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-03 01:22:03,832 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 90.86 2021-05-03 01:22:06,549 | test | INFO | pool-4-thread-29 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2647 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:06,551 | test | INFO | pool-4-thread-29 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:07,138 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oQ81gC-45-31000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-510 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-650 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-670 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:07,230 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:0%pF3gCpSLoIme4-36-413000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2559 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:07,232 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:0%pF3gCpSLoIme4-36-413000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2559 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:08,295 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-511 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-670 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:09,078 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 92.83 2021-05-03 01:22:09,289 | test | INFO | pool-4-thread-10 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2592 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:09,289 | test | INFO | pool-4-thread-10 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2592 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:09,293 | test | INFO | pool-4-thread-23 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oQ81gC-45-31000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2592 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:09,295 | test | INFO | pool-4-thread-23 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oQ81gC-45-31000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2592 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:09,298 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2756 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:09,299 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2756 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:11,038 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oZVzxffl7W3w8vsfLqcPn-45-27000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-635 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-511 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-670 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:13,506 | test | INFO | pool-4-thread-22 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oQ81gC-45-31000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-102 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-91 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:13,513 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-102 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-91 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:14,279 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 94.58 2021-05-03 01:22:17,453 | test | INFO | pool-4-thread-25 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:17,463 | test | INFO | pool-4-thread-26 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:uylFiogRHveaj-45-27000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:18,684 | test | INFO | pool-4-thread-14 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:uylFiogRHveaj-45-27000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-234 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-251 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:18,698 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-234 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:19,513 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 96.75 2021-05-03 01:22:24,118 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oZVzxffl7W3w8vsfLqcPn-45-27000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-39 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-107 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:24,128 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-39 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:24,200 | test | INFO | pool-4-thread-5 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-546 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-553 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-675 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:24,743 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 98.64 2021-05-03 01:22:29,967 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 01:22:30,043 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:0%pF3gCpSLoIme4-36-413000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:30,043 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:0%pF3gCpSLoIme4-36-413000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2615 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2785 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:32,066 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:32,066 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2836 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:32,700 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:32,704 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oQ81gC-45-31000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:32,710 | test | INFO | pool-4-thread-29 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2723 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:32,710 | test | INFO | pool-4-thread-29 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2723 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:32,713 | test | INFO | pool-4-thread-15 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oZVzxffl7W3w8vsfLqcPn-45-27000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-882 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:35,135 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 01:22:35,920 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:35,921 | test | INFO | pool-4-thread-20 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2628 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:37,910 | test | INFO | pool-4-thread-14 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:uylFiogRHveaj-45-27000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-465 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:37,920 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-465 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:42,115 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 01:22:45,119 | test | INFO | pool-4-thread-5 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:52,588 | test | INFO | pool-4-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9bd6 type=client open_count=1 channel=[id: 0xe0060eff, 0.0.0.0/0.0.0.0:51399] timeout=300.0>) 2021-05-03 01:22:52,723 | test | ERROR | pool-4-thread-9 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 01:22:53,132 | test | INFO | pool-4-thread-22 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:oQ81gC-45-31000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-253 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:53,148 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-253 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:22:55,325 | test | INFO | pool-4-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 264.720999956 sec 2021-05-03 01:23:11,042 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 01:23:11,066 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 01:24:08,150 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-03 01:24:36,931 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 01:24:36,934 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 01:24:36,937 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 01:24:36,937 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 4 docs from 0 to 5000 2021-05-03 01:24:37,048 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-03 01:24:37,049 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 5 docs from 0 to 500 2021-05-03 01:24:37,049 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 01:24:37,051 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 0 to 500 2021-05-03 01:24:37,051 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 01:24:37,052 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 500 to 980 2021-05-03 01:24:37,052 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 500 to 980 2021-05-03 01:24:37,052 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 0 to 500 2021-05-03 01:24:37,053 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 01:24:37,055 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 6 docs from 0 to 500 2021-05-03 01:24:37,055 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 01:24:37,056 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-03 01:24:37,056 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-03 01:24:37,058 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 01:24:37,059 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 01:24:37,059 | test | ERROR | pool-4-thread-17 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 01:30:16,279 | test | INFO | pool-4-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **********************************************************************************************************************************************....*...**.....*.....*...**.*X | 81.812 | | 74.412 | ..................X | 74.412 | | 70.700 | .................X | 70.700 | | 64.100 | ...............*X | 64.100 | | 60.900 | ..............X | 60.900 | | 59.400 | ..............*X | 59.400 | | 57.800 | ..............**.***X | 64.800 | | 57.700 | ..............X | 57.700 | | 52.700 | ............**..**...*...**X | 83.100 | | 81.900 | ....................X | 81.900 | | 75.200 | ..................*X | 75.200 | | 68.000 | ................*X | 68.000 | | 66.000 | ................X | 66.000 | | 59.689 | ..............****..*X | 66.816 | | 65.416 | ................*X | 65.416 | | 62.702 | ...............*.**..*X | 75.386 | | 75.114 | ..................X | 75.114 | | 66.914 | ................*X | 66.914 | | 60.914 | ..............*..*.**.*X | 74.700 | | 67.500 | ................*.*X | 69.400 | | 56.200 | .............*X | 56.200 | | 43.500 | ..........X | 43.500 | | 32.400 | .......***...*.*X | 49.500 | | 43.600 | ..........*X | 43.600 | | 23.700 | .....X | 23.700 | | 7.400 | *X | 7.400 | | 0.000 | *******************...*....**........**..........***.......*..........*....*X | 178.563 | | 160.063 | ........................................*X | 160.063 | | 120.596 | ..............................X | 120.596 | | 112.096 | ............................**...**............***.....*...........X | 231.921 | | 229.721 | ..........................................................**.**....*......**..........*X | 310.123 | | 301.971 | .............................................................................*......X | 323.950 | | 307.850 | ...............................................................................**X | 307.850 | | 270.324 | .....................................................................**X | 270.324 | | 260.824 | ..................................................................*.............*.........**........*X | 375.741 | | 352.162 | ..........................................................................................X | 352.162 | | 333.600 | .....................................................................................**........*X | 364.200 | | 363.200 | .............................................................................................*......X | 384.000 | | 319.800 | ..................................................................................*X | 319.800 | | 306.300 | ..............................................................................*X | 306.300 | | 262.500 | ...................................................................X | 262.500 | +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 01:30:17,267 | test | INFO | pool-4-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........*....**........*....*...*X | 483.107 | | 417.821 | .......................*X | 417.821 | | 372.628 | ....................*...*...................**............*.*X | 962.446 | | 822.126 | ...............................................**.**.....*.*...***.......*...........................**........X | 1712.171 | | 1287.544 | ..........................................................................*X | 1287.544 | | 1063.422 | .............................................................X | 1063.422 | | 894.222 | ...................................................X | 894.222 | | 412.746 | .......................*X | 412.746 | | 334.483 | ..................*.**..*..*.X | 436.574 | | 415.400 | .......................*X | 415.400 | | 398.765 | ......................X | 398.765 | | 334.965 | ..................X | 334.965 | | 314.465 | .................**.*X | 326.320 | | 301.620 | ................X | 301.620 | | 276.026 | ...............*X | 276.026 | | 221.149 | ...........**X | 221.849 | | 179.245 | .........**.*....**.*X | 285.064 | | 205.680 | ...........*X | 205.680 | | 189.195 | ..........X | 189.195 | | 169.495 | ........X | 169.495 | | 142.491 | .......X | 142.491 | | 137.825 | .......**..**.......*........*.....X | 516.989 | | 465.312 | ..........................X | 465.312 | | 375.886 | ....................X | 375.886 | | 289.778 | ...............X | 289.778 | | 253.287 | .............*X | 253.287 | | 205.571 | ...........*X | 205.571 | | 160.571 | ........X | 160.571 | | 129.621 | ......X | 129.621 | | 86.621 | ....*.**....*....X | 245.797 | | 236.744 | ............X | 236.744 | | 199.546 | ..........X | 199.546 | | 191.846 | ..........**....*....**X | 339.027 | | 305.527 | ................**.*..*.....**.....*.........**....*.....X | 822.300 | | 784.800 | ............................................**..X | 817.800 | | 781.200 | ............................................X | 781.200 | | 762.800 | ...........................................*X | 762.800 | | 693.090 | .......................................*X | 693.090 | | 683.190 | ......................................*..**.X | 732.517 | | 726.017 | .........................................X | 726.017 | | 708.417 | ........................................*X | 708.417 | | 679.700 | ......................................*X | 679.700 | | 587.714 | .................................***....*......*............X | 965.085 | | 943.885 | ......................................................*X | 943.885 | | 892.805 | ...................................................X | 892.805 | | 728.157 | .........................................*X | 728.157 | | 700.057 | .......................................*X | 700.057 | | 679.310 | ......................................X | 679.310 | | 669.032 | ......................................X | 669.032 | | 583.132 | .................................*X | 583.132 | | 410.158 | ......................X | 410.158 | | 392.588 | .....................**..*X | 427.188 | | 414.900 | .......................*X | 414.900 | | 318.100 | .................*X | 318.100 | | 292.000 | ................*.**.....X | 406.300 | | 395.400 | ......................*..*X | 437.100 | | 397.500 | ......................**.X | 419.500 | | 416.900 | .......................X | 416.900 | | 387.800 | .....................X | 387.800 | | 387.800 | .....................X | 387.800 | | 380.100 | .....................X | 380.100 | | 285.000 | ...............**...**.....**X | 422.300 | | 364.900 | ....................X | 364.900 | | 320.600 | .................X | 320.600 | | 315.712 | .................**...*X | 361.612 | | 338.312 | ..................X | 338.312 | | 318.374 | .................*X | 318.374 | | 281.874 | ...............**.....*....*...*X | 484.979 | | 403.964 | ......................X | 403.964 | | 313.964 | .................*X | 313.964 | | 228.764 | ............**......*....*.........*X | 554.000 | | 527.000 | .............................X | 527.000 | | 512.400 | ............................*X | 512.400 | | 428.500 | ........................*X | 428.500 | | 361.300 | ....................***X | 375.500 | | 347.600 | ...................X | 347.600 | | 325.700 | ..................**...**.................*.......**..X | 829.541 | | 719.841 | .........................................*...X | 780.441 | | 780.441 | ............................................*.......**.........*..X | 1082.243 | | 1076.853 | .............................................................***.*.X | 1103.867 | | 1087.842 | ..............................................................*X | 1087.842 | | 1043.242 | ...........................................................X | 1043.242 | | 1040.261 | ...........................................................**..*X | 1071.629 | | 999.829 | .........................................................X | 999.829 | | 831.352 | ...............................................X | 831.352 | | 594.952 | .................................*X | 594.952 | | 375.452 | ....................*X | 375.452 | | 265.300 | ..............X | 265.300 | | 185.800 | .........*X | 185.800 | | 178.876 | .........X | 178.876 | | 154.876 | ........*X | 154.876 | | 134.376 | ......*X | 134.376 | | 99.099 | ....X | 99.099 | | 59.099 | ..X | 59.099 | | 38.697 | .*X | 38.697 | | 25.698 | *X | 25.698 | | 8.898 | X | 8.898 | | 3.100 | **X | 3.100 | | 0.000 | **********************************X | 0.000 | +----------+------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 01:30:17,551 | test | INFO | pool-4-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****X | 0.700 | | 0.400 | X | 0.400 | | 0.200 | *X | 0.200 | | 0.200 | ***X | 0.200 | | 0.000 | *******************....*....**........*.....**...*..**.....**..........*X | 144.231 | | 140.000 | ......................................X | 140.000 | | 134.382 | .....................................X | 134.382 | | 106.682 | .............................X | 106.682 | | 93.882 | .........................**.**.X | 98.616 | | 92.460 | .........................X | 92.460 | | 76.962 | ....................*...**.*........X | 115.944 | | 101.244 | ...........................X | 101.244 | | 82.900 | ......................X | 82.900 | | 51.900 | .............*...*...............*..........**............*.*.X | 199.385 | | 186.136 | ....................................................X | 186.136 | | 143.336 | .......................................X | 143.336 | | 131.336 | ....................................*X | 131.336 | | 109.970 | ..............................*...................**.....X | 194.237 | | 180.221 | ..................................................X | 180.221 | | 159.104 | ............................................*.**......*X | 184.366 | | 179.266 | ..................................................X | 179.266 | | 166.345 | ..............................................X | 166.345 | | 153.345 | ..........................................**.......*..*.............X | 230.113 | | 209.510 | ..........................................................*...............**......*.................*X | 343.337 | | 293.326 | ..................................................................................*X | 293.326 | | 290.926 | ..................................................................................**...........*......*X | 350.490 | | 276.090 | .............................................................................X | 276.090 | | 156.690 | ...........................................*X | 156.690 | | 48.600 | ............X | 48.600 | | 4.200 | *X | 4.200 | | 0.000 | *****************..*..*.....**.......X | 57.897 | | 53.801 | ..............*X | 53.801 | | 33.301 | ........X | 33.301 | | 4.000 | X | 4.000 | | 2.799 | **.*X | 4.799 | | 3.800 | ******X | 5.300 | | 2.800 | *X | 2.800 | | 1.700 | X | 1.700 | | 1.000 | X | 1.000 | | 0.300 | *X | 0.300 | | 0.000 | *********************************************************************************************************************************************X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 01:30:17,813 | test | WARNING | Thread-5502 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:5QOcV1hUe-36-568000:5NrUP-36-691000 create failed keys from task: [u'test_collections-4086', u'test_collections-4408', u'test_collections-4116', u'test_collections-4028', u'test_collections-4598'] 2021-05-03 01:30:17,815 | test | WARNING | Thread-5503 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1RiYR3nGoHj8pWbFI-36-279000:A-36-412000 create failed keys from task: [u'test_collections-2491', u'test_collections-1816', u'test_collections-2501', u'test_collections-2756', u'test_collections-1986', u'test_collections-2836', u'test_collections-2470', u'test_collections-2343', u'test_collections-2223'] 2021-05-03 01:30:17,815 | test | WARNING | Thread-5504 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1RiYR3nGoHj8pWbFI-36-279000:0%pF3gCpSLoIme4-36-413000 create failed keys from task: [u'test_collections-2615', u'test_collections-1757', u'test_collections-2559', u'test_collections-1714', u'test_collections-1684', u'test_collections-2785'] 2021-05-03 01:30:17,815 | test | WARNING | Thread-5505 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:DCdXwh-36-882000:ETs0-36-992000 create failed keys from task: [u'test_collections-4080', u'test_collections-4474', u'test_collections-4110'] 2021-05-03 01:30:17,831 | test | WARNING | Thread-5510 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 update failed keys from task: [u'test_collections-816', u'test_collections-849', u'test_collections-632', u'test_collections-786', u'test_collections-883', u'test_collections-565', u'test_collections-543', u'test_collections-510', u'test_collections-562', u'test_collections-670'] 2021-05-03 01:30:17,831 | test | WARNING | Thread-5509 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:uylFiogRHveaj-45-27000 update failed keys from task: [u'test_collections-816', u'test_collections-849', u'test_collections-632', u'test_collections-786', u'test_collections-883', u'test_collections-543', u'test_collections-510', u'test_collections-670'] 2021-05-03 01:30:17,832 | test | WARNING | Thread-5512 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:oQ81gC-45-31000 update failed keys from task: [u'test_collections-519', u'test_collections-896', u'test_collections-883', u'test_collections-510', u'test_collections-650', u'test_collections-670'] 2021-05-03 01:30:17,832 | test | WARNING | Thread-5506 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:DCdXwh-36-882000:3c3l6Bro-36-991000 create failed keys from task: [u'test_collections-4080', u'test_collections-3725', u'test_collections-4474', u'test_collections-4110'] 2021-05-03 01:30:17,832 | test | WARNING | Thread-5513 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:oZVzxffl7W3w8vsfLqcPn-45-27000 update failed keys from task: [u'test_collections-506', u'test_collections-635', u'test_collections-511', u'test_collections-883', u'test_collections-510', u'test_collections-882', u'test_collections-670'] 2021-05-03 01:30:17,990 | test | WARNING | Thread-5512 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:oQ81gC-45-31000 delete failed keys from task: [u'test_collections-216', u'test_collections-102', u'test_collections-91', u'test_collections-147', u'test_collections-386', u'test_collections-253'] 2021-05-03 01:30:18,005 | test | WARNING | Thread-5512 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:oQ81gC-45-31000 create failed keys from task: [u'test_collections-2592'] 2021-05-03 01:30:18,015 | test | WARNING | Thread-5509 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:uylFiogRHveaj-45-27000 delete failed keys from task: [u'test_collections-102', u'test_collections-234', u'test_collections-465', u'test_collections-140', u'test_collections-251'] 2021-05-03 01:30:18,023 | test | WARNING | Thread-5510 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 delete failed keys from task: [u'test_collections-214', u'test_collections-102', u'test_collections-234', u'test_collections-465', u'test_collections-384', u'test_collections-140'] 2021-05-03 01:30:18,030 | test | WARNING | Thread-5513 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:oZVzxffl7W3w8vsfLqcPn-45-27000 delete failed keys from task: [u'test_collections-39', u'test_collections-94', u'test_collections-107', u'test_collections-5'] 2021-05-03 01:30:18,039 | test | WARNING | Thread-5510 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:quJ17M9yjy59qtdY3d4-45-27000 create failed keys from task: [u'test_collections-2723', u'test_collections-2647'] 2021-05-03 01:30:18,062 | test | WARNING | Thread-5654 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:KY8-35-122000:En7vNv5jc-35-226000 create failed keys from task: [u'test_collections-85', u'test_collections-761', u'test_collections-461', u'test_collections-231'] 2021-05-03 01:30:18,062 | test | WARNING | Thread-5656 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:hcchR-34-251000:fSvY0hSTCqB0_h5DUD-34-458000 create failed keys from task: [u'test_collections-143'] 2021-05-03 01:30:18,063 | test | WARNING | Thread-5658 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:IUOrBu8kMdcThx-34-789000:Kmw1IEJXqudQA-34-976000 create failed keys from task: [u'test_collections-733', u'test_collections-30', u'test_collections-20', u'test_collections-157', u'test_collections-532', u'test_collections-182'] 2021-05-03 01:30:18,065 | test | WARNING | Thread-5659 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:IUOrBu8kMdcThx-34-789000:Q4eu7PwU8-34-977000 create failed keys from task: [u'test_collections-59', u'test_collections-408', u'test_collections-923', u'test_collections-448', u'test_collections-598', u'test_collections-1064'] 2021-05-03 01:30:18,134 | test | WARNING | Thread-5501 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:5QOcV1hUe-36-568000:pb-36-692000 create failed keys from task: [u'test_collections-4605', u'test_collections-4795'] 2021-05-03 01:30:18,062 | test | WARNING | Thread-5657 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:hcchR-34-251000:WPAWrA0J-34-457000 create failed keys from task: [u'test_collections-19', u'test_collections-85', u'test_collections-92', u'test_collections-761', u'test_collections-461', u'test_collections-231'] 2021-05-03 01:30:18,239 | test | WARNING | Thread-5511 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 update failed keys from task: [u'test_collections-808', u'test_collections-506', u'test_collections-511', u'test_collections-883', u'test_collections-510', u'test_collections-670'] 2021-05-03 01:30:18,269 | test | WARNING | Thread-5514 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 update failed keys from task: [u'test_collections-546', u'test_collections-854', u'test_collections-553', u'test_collections-675'] 2021-05-03 01:30:18,302 | test | WARNING | Thread-5511 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 delete failed keys from task: [u'test_collections-216', u'test_collections-102', u'test_collections-91', u'test_collections-147', u'test_collections-386', u'test_collections-253'] 2021-05-03 01:30:18,339 | test | WARNING | Thread-5511 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:cZg_7ezj8L1Lg-wl-45-28000 create failed keys from task: [u'test_collections-2592'] 2021-05-03 01:30:18,342 | test | WARNING | Thread-5514 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 delete failed keys from task: [u'test_collections-39', u'test_collections-94'] 2021-05-03 01:30:18,351 | test | WARNING | Thread-5514 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:IuL1eIx-45-26000:thNzFzq_8IADOySbNqvY-45-29000 create failed keys from task: [u'test_collections-2628'] 2021-05-03 01:30:30,890 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 01:31:43,171 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 01:31:43,171 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 01:32:05,236 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 01:32:05,237 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-03 01:32:25,885 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 01:32:25,887 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 01:33:09,619 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 01:33:29,154 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1610000134 seconds to complete 2021-05-03 01:33:29,174 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok During the test, ---------------------------------------------------------------------- Ran 1 test in 2049.028s OK Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_2 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_3', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 3, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 01:33:39,102 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 01:33:40,069 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 01:33:40,628 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba583 type=client open_count=1 channel=[id: 0x16bc4810, 0.0.0.0/0.0.0.0:47015] timeout=300.0>) 2021-05-03 01:33:46,121 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba584 type=client open_count=1 channel=[id: 0xed073881, 0.0.0.0/0.0.0.0:35276] timeout=300.0>) 2021-05-03 01:33:48,147 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba585 type=client open_count=1 channel=[id: 0xf0b3893b, 0.0.0.0/0.0.0.0:35557] timeout=300.0>) 2021-05-03 01:33:50,173 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba586 type=client open_count=1 channel=[id: 0xf89512ad, 0.0.0.0/0.0.0.0:43501] timeout=300.0>) 2021-05-03 01:33:52,201 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba587 type=client open_count=1 channel=[id: 0x17721f17, 0.0.0.0/0.0.0.0:47662] timeout=300.0>) 2021-05-03 01:33:54,223 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba588 type=client open_count=1 channel=[id: 0x01974924, 0.0.0.0/0.0.0.0:40733] timeout=300.0>) 2021-05-03 01:33:56,246 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba589 type=client open_count=1 channel=[id: 0x729beda0, 0.0.0.0/0.0.0.0:41026] timeout=300.0>) 2021-05-03 01:33:58,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba58a type=client open_count=1 channel=[id: 0x78b1c2c7, 0.0.0.0/0.0.0.0:47166] timeout=300.0>) 2021-05-03 01:34:00,295 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xba58b type=client open_count=1 channel=[id: 0xd45efef4, 0.0.0.0/0.0.0.0:47995] timeout=300.0>) 2021-05-03 01:34:02,319 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba58c type=client open_count=1 channel=[id: 0x2cba6c24, 0.0.0.0/0.0.0.0:58156] timeout=300.0>) 2021-05-03 01:34:10,413 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba58d type=client open_count=1 channel=[id: 0xcba88bfe, 0.0.0.0/0.0.0.0:41158] timeout=300.0>) 2021-05-03 01:34:12,437 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba58e type=client open_count=1 channel=[id: 0x5de8a30b, 0.0.0.0/0.0.0.0:53087] timeout=300.0>) 2021-05-03 01:34:14,463 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba58f type=client open_count=1 channel=[id: 0x8486151b, 0.0.0.0/0.0.0.0:35204] timeout=300.0>) 2021-05-03 01:34:16,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba590 type=client open_count=1 channel=[id: 0x0e56237c, 0.0.0.0/0.0.0.0:33222] timeout=300.0>) 2021-05-03 01:34:18,509 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba591 type=client open_count=1 channel=[id: 0x47cc5e18, 0.0.0.0/0.0.0.0:55221] timeout=300.0>) 2021-05-03 01:34:20,530 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba592 type=client open_count=1 channel=[id: 0x823416d2, 0.0.0.0/0.0.0.0:39522] timeout=300.0>) 2021-05-03 01:34:22,553 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba593 type=client open_count=1 channel=[id: 0xa34e7535, 0.0.0.0/0.0.0.0:46574] timeout=300.0>) 2021-05-03 01:34:24,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba594 type=client open_count=1 channel=[id: 0x55594bb5, 0.0.0.0/0.0.0.0:56173] timeout=300.0>) 2021-05-03 01:34:26,607 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xba595 type=client open_count=1 channel=[id: 0x757d250e, 0.0.0.0/0.0.0.0:46154] timeout=300.0>) 2021-05-03 01:34:28,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba596 type=client open_count=1 channel=[id: 0x31dc7179, 0.0.0.0/0.0.0.0:52665] timeout=300.0>) 2021-05-03 01:34:36,109 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba597 type=client open_count=1 channel=[id: 0x4ab17978, 0.0.0.0/0.0.0.0:58193] timeout=300.0>) 2021-05-03 01:34:38,134 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba598 type=client open_count=1 channel=[id: 0x46ac4423, 0.0.0.0/0.0.0.0:57287] timeout=300.0>) 2021-05-03 01:34:40,157 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba599 type=client open_count=1 channel=[id: 0xc6b0f6e7, 0.0.0.0/0.0.0.0:42187] timeout=300.0>) 2021-05-03 01:34:42,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba59a type=client open_count=1 channel=[id: 0x6344ce0a, 0.0.0.0/0.0.0.0:39605] timeout=300.0>) 2021-05-03 01:34:44,204 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba59b type=client open_count=1 channel=[id: 0x800371db, 0.0.0.0/0.0.0.0:56375] timeout=300.0>) 2021-05-03 01:34:46,230 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba59c type=client open_count=1 channel=[id: 0xfbdeb3dd, 0.0.0.0/0.0.0.0:57033] timeout=300.0>) 2021-05-03 01:34:48,256 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba59d type=client open_count=1 channel=[id: 0x3b52d4b9, 0.0.0.0/0.0.0.0:53547] timeout=300.0>) 2021-05-03 01:34:50,279 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba59e type=client open_count=1 channel=[id: 0x731e94f6, 0.0.0.0/0.0.0.0:44343] timeout=300.0>) 2021-05-03 01:34:52,301 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xba59f type=client open_count=1 channel=[id: 0xbfd575d2, 0.0.0.0/0.0.0.0:44209] timeout=300.0>) 2021-05-03 01:34:54,322 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a0 type=client open_count=1 channel=[id: 0xc7cbbbb8, 0.0.0.0/0.0.0.0:49118] timeout=300.0>) 2021-05-03 01:35:02,424 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a1 type=client open_count=1 channel=[id: 0x39ad2f89, 0.0.0.0/0.0.0.0:34762] timeout=300.0>) 2021-05-03 01:35:04,448 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a2 type=client open_count=1 channel=[id: 0x5f3ca467, 0.0.0.0/0.0.0.0:44355] timeout=300.0>) 2021-05-03 01:35:06,471 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a3 type=client open_count=1 channel=[id: 0x56463751, 0.0.0.0/0.0.0.0:35117] timeout=300.0>) 2021-05-03 01:35:08,497 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a4 type=client open_count=1 channel=[id: 0xfac3d44c, 0.0.0.0/0.0.0.0:34925] timeout=300.0>) 2021-05-03 01:35:10,522 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a5 type=client open_count=1 channel=[id: 0x52834837, 0.0.0.0/0.0.0.0:46749] timeout=300.0>) 2021-05-03 01:35:12,546 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a6 type=client open_count=1 channel=[id: 0x46e16780, 0.0.0.0/0.0.0.0:51798] timeout=300.0>) 2021-05-03 01:35:14,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a7 type=client open_count=1 channel=[id: 0x57dc5f65, 0.0.0.0/0.0.0.0:50310] timeout=300.0>) 2021-05-03 01:35:16,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a8 type=client open_count=1 channel=[id: 0xce6f5672, 0.0.0.0/0.0.0.0:48542] timeout=300.0>) 2021-05-03 01:35:18,622 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xba5a9 type=client open_count=1 channel=[id: 0x3ef8424e, 0.0.0.0/0.0.0.0:54244] timeout=300.0>) 2021-05-03 01:35:20,651 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xba5aa type=client open_count=1 channel=[id: 0x96db56da, 0.0.0.0/0.0.0.0:56332] timeout=300.0>) 2021-05-03 01:35:26,500 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xba5ab type=client open_count=1 channel=[id: 0xaaf796ba, 0.0.0.0/0.0.0.0:35503] timeout=300.0>) 2021-05-03 01:35:28,535 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xba5ac type=client open_count=1 channel=[id: 0xd9964fa3, 0.0.0.0/0.0.0.0:33239] timeout=300.0>) 2021-05-03 01:35:30,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xba5ad type=client open_count=1 channel=[id: 0x821a771b, 0.0.0.0/0.0.0.0:44489] timeout=300.0>) 2021-05-03 01:35:32,617 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xba5ae type=client open_count=1 channel=[id: 0x18029fee, 0.0.0.0/0.0.0.0:48262] timeout=300.0>) 2021-05-03 01:35:34,677 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xba5af type=client open_count=1 channel=[id: 0x99ed3fe2, 0.0.0.0/0.0.0.0:49298] timeout=300.0>) 2021-05-03 01:35:36,723 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 01:35:57,411 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 01:36:13,289 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 01:36:13,381 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 01:36:13,559 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 01:36:47,618 | test | INFO | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 7.31768861973 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 01:36:47,644 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 01:36:57,749 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 01:36:57,822 | test | INFO | pool-5-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2049999237 sec 2021-05-03 01:36:58,658 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 01:37:42,069 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 01:37:42,351 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 636835136 | 71159810 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 569571936 | 95433452 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 665388384 | 51003500 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-03 01:37:42,351 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 01:38:38,894 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 01:38:38,910 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 01:53:25,121 | test | INFO | pool-5-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....**....**....X | 24.200 | | 24.191 | ...........X | 24.191 | | 17.591 | ........*X | 17.591 | | 8.791 | ...X | 8.791 | | 0.000 | *********************************.....**............................*........................................................**...........*X | 192.262 | | 30.300 | ..............**X | 30.300 | | 0.000 | *******************************************...***X | 7.500 | | 5.500 | .X | 5.500 | | 1.500 | *X | 1.500 | | 0.000 | ************************************************************************************************************************************************************************************......*..........***..X | 35.700 | | 25.700 | ............X | 25.700 | | 5.700 | .*X | 5.700 | | 0.000 | ********************************************************************************************..*......**..X | 19.496 | | 14.296 | ......X | 14.296 | | 3.896 | .*X | 3.896 | | 0.000 | ************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 01:53:26,573 | test | INFO | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +----------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..**................**............................................................................**......*X | 1181.067 | | 1088.467 | ...........................................................................................X | 1088.467 | | 910.328 | ............................................................................*X | 910.328 | | 872.528 | ........................................................................*..*.*X | 898.556 | | 897.956 | ...........................................................................*X | 897.956 | | 888.756 | ..........................................................................X | 888.756 | | 854.287 | .......................................................................**..X | 878.687 | | 871.087 | ........................................................................X | 871.087 | | 845.300 | ......................................................................*X | 845.300 | | 778.100 | ................................................................*X | 778.100 | | 604.800 | ..................................................*X | 604.800 | | 579.600 | ................................................*X | 579.600 | | 570.700 | ...............................................X | 570.700 | | 522.000 | ...........................................*X | 522.000 | | 514.100 | ..........................................***X | 517.636 | | 508.311 | ..........................................***.....**.*X | 580.097 | | 549.697 | .............................................X | 549.697 | | 497.238 | .........................................***.......*.....**.*X | 657.992 | | 620.557 | ...................................................X | 620.557 | | 611.641 | ..................................................X | 611.641 | | 603.003 | ..................................................**X | 603.230 | | 594.330 | .................................................**.*..*...**.*X | 679.216 | | 621.330 | ...................................................**X | 621.330 | | 568.763 | ...............................................X | 568.763 | | 529.063 | ...........................................X | 529.063 | | 424.609 | ..................................*X | 424.609 | | 414.509 | ..................................***.**.***...*.**..*...**....**.*...X | 647.100 | | 633.300 | ....................................................***.**.*X | 653.267 | | 643.267 | .....................................................X | 643.267 | | 603.371 | ..................................................*X | 603.371 | | 591.671 | .................................................**...*X | 629.678 | | 628.678 | ....................................................*****.X | 640.818 | | 600.918 | .................................................X | 600.918 | | 534.978 | ............................................*X | 534.978 | | 509.030 | ..........................................*X | 509.030 | | 486.330 | ........................................X | 486.330 | | 442.915 | ....................................X | 442.915 | | 420.400 | ..................................**..**..X | 464.600 | | 462.200 | ......................................*X | 462.200 | | 449.710 | .....................................X | 449.710 | | 448.710 | ....................................**...**.......**...*X | 602.042 | | 591.820 | .................................................*.*X | 612.220 | | 569.700 | ...............................................***..**...X | 633.989 | | 614.629 | ...................................................X | 614.629 | | 529.171 | ...........................................*X | 529.171 | | 466.871 | ......................................*X | 466.871 | | 449.972 | .....................................X | 449.972 | | 444.634 | ....................................***X | 444.634 | | 442.301 | ....................................*.*..**..*...**.**...........***..*.X | 711.862 | | 683.834 | ........................................................*X | 683.834 | | 683.434 | ........................................................X | 683.434 | | 676.756 | ........................................................**X | 677.677 | | 663.977 | .......................................................*X | 663.977 | | 574.648 | ...............................................*X | 574.648 | | 531.114 | ...........................................*X | 531.114 | | 491.314 | ........................................X | 491.314 | | 416.366 | ..................................**.**..*.....X | 516.938 | | 510.800 | ..........................................**.****..*....**...**..**.*X | 672.300 | | 651.700 | ......................................................*X | 651.700 | | 597.683 | .................................................X | 597.683 | | 526.583 | ...........................................*X | 526.583 | | 505.783 | .........................................X | 505.783 | | 472.615 | .......................................*.*X | 488.290 | | 472.190 | ......................................X | 472.190 | | 462.451 | ......................................****.......***.*...*.....**...**.**X | 699.289 | | 671.589 | .......................................................X | 671.589 | | 653.100 | ......................................................X | 653.100 | | 644.690 | .....................................................*X | 644.690 | | 634.390 | ....................................................*X | 634.390 | | 527.287 | ...........................................*X | 527.287 | | 489.887 | ........................................*X | 489.887 | | 459.887 | .....................................*X | 459.887 | | 426.680 | ...................................***.*....***.**....*...**..**X | 610.726 | | 607.980 | ..................................................*X | 607.980 | | 584.840 | ................................................*.X | 596.960 | | 591.984 | .................................................**..X | 614.771 | | 602.529 | ..................................................**X | 603.640 | | 569.692 | ...............................................X | 569.692 | | 431.410 | ...................................*X | 431.410 | | 322.524 | ..........................*X | 322.524 | | 199.024 | ...............X | 199.024 | | 42.824 | ..*X | 42.824 | | 2.500 | **........***.........*...............*..................**...*X | 633.500 | | 625.981 | ....................................................***..*.X | 672.935 | | 651.435 | ......................................................*X | 651.435 | | 617.880 | ...................................................*X | 617.880 | | 515.957 | ..........................................*X | 515.957 | | 503.963 | .........................................**X | 506.263 | | 501.637 | .........................................**X | 502.886 | | 492.886 | ........................................**.X | 505.700 | | 490.500 | ........................................**..*..*...*X | 577.423 | | 568.056 | ...............................................**X | 574.156 | | 549.997 | .............................................*X | 549.997 | | 542.528 | ............................................X | 542.528 | | 521.418 | ...........................................*..**..*..X | 593.913 | | 567.750 | ...............................................*X | 567.750 | | 522.627 | ...........................................X | 522.627 | | 448.915 | .....................................*X | 448.915 | | 427.930 | ...................................**.**.**X | 451.477 | | 448.777 | ....................................**..**..*.....*..***.****..*......*..*X | 701.313 | | 685.279 | .........................................................X | 685.279 | | 632.429 | ....................................................*X | 632.429 | | 603.029 | ..................................................X | 603.029 | | 567.564 | ...............................................X | 567.564 | | 517.339 | ..........................................*X | 517.339 | | 488.639 | ........................................**.*...*.*X | 549.664 | | 542.572 | ............................................X | 542.572 | | 492.226 | ........................................**.**...*......X | 606.229 | | 601.700 | .................................................*X | 601.700 | | 594.000 | .................................................*...**...**....X | 710.800 | | 630.500 | ....................................................X | 630.500 | | 571.896 | ...............................................*X | 571.896 | | 459.996 | .....................................X | 459.996 | | 358.996 | .............................*X | 358.996 | | 351.097 | ............................**...........**..**...*.......*....*X | 662.700 | | 657.700 | ......................................................X | 657.700 | | 597.971 | .................................................*X | 597.971 | +----------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 01:53:27,197 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ********************************************************************************************************************************************************************************************************************************************************************************************.....*...............*......*X | 54.400 | | 49.596 | .......................*X | 49.596 | | 17.596 | .......X | 17.596 | | 5.596 | .*X | 5.596 | | 0.000 | *******************************************************************************************....**........**.........................*......................................................*.........*X | 203.029 | | 162.829 | ...............................................................................X | 162.829 | | 45.029 | .....................*X | 45.029 | | 12.300 | .....*X | 12.300 | | 0.400 | X | 0.400 | | 0.000 | *****X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 01:53:28,032 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 61.4814814815 | 3693387776 | 2564141056 | 0 / 3909087232 | 126400 / 254866 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 35.6808731809 | 3693387776 | 2639011840 | 0 / 3909087232 | 125095 / 252729 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 37.0722922562 | 6078930944 | 4792360960 | 0 / 3758092288 | 126501 / 246400 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 49.7924234561 | 4201836544 | 3094106112 | 0 / 3758092288 | 126092 / 253908 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 01:53:42,740 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 01:55:33,039 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155579024 | 241168238 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 156176592 | 258994148 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 434346656 | 467132098 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-03 01:55:33,042 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 01:55:44,193 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during swap_rebalance 2021-05-03 01:55:44,194 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : swap_rebalance 2021-05-03 01:55:51,924 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 01:55:52,180 | test | INFO | pool-5-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 20.8985360929 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 20.8099847173 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 13.38081913 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 1.15115115115 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 13.7164750958 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 01:55:52,276 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 01:55:57,638 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 01:56:04,221 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 1.11 2021-05-03 01:56:05,019 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 01:56:09,480 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 3.93 2021-05-03 01:56:14,648 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 6.71 2021-05-03 01:56:19,871 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 9.67 2021-05-03 01:56:25,029 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-128 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:25,032 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-128 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:25,061 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 12.35 2021-05-03 01:56:26,927 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-327 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:26,928 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-327 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:28,023 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-540 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:28,023 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-540 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:30,221 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 14.92 2021-05-03 01:56:30,253 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:30,255 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-667 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:35,378 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:35,378 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1318 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1288 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:35,410 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 17.73 2021-05-03 01:56:36,293 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:36,295 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1570 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,576 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,578 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,584 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,585 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,588 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,585 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,588 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:37,589 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-192 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:39,528 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1771 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:39,529 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1771 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:40,970 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 20.72 2021-05-03 01:56:45,426 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket1:6ADiXxcxF2QZ_0H-1-771000:v2oMI3Sg_P35%5lnm-2-29000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-33 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,427 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket1:6ADiXxcxF2QZ_0H-1-771000:v2oMI3Sg_P35%5lnm-2-29000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-33 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,815 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,815 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,816 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,815 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,816 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,818 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,819 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:45,818 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:46,130 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 23.33 2021-05-03 01:56:47,766 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2092 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2102 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:47,767 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2092 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2155 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2102 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:51,276 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 26.03 2021-05-03 01:56:51,312 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in bucket1:6ADiXxcxF2QZ_0H-1-771000:t5jzAWJRmG_s6-2-52000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:51,313 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in bucket1:6ADiXxcxF2QZ_0H-1-771000:t5jzAWJRmG_s6-2-52000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:51,326 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket1:6ADiXxcxF2QZ_0H-1-771000:v2oMI3Sg_P35%5lnm-2-29000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:51,328 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket1:6ADiXxcxF2QZ_0H-1-771000:v2oMI3Sg_P35%5lnm-2-29000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:56,494 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 28.66 2021-05-03 01:56:58,408 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1321 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:58,410 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1321 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:58,414 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1321 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:56:58,415 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1321 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,339 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,341 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,342 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,345 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,348 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,351 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,359 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,361 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:01,694 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 31.48 2021-05-03 01:57:04,980 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:04,980 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:04,983 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:04,984 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:04,986 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:04,986 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:04,987 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:04,987 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1639 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:06,795 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 01:57:11,904 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 01:57:19,881 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 01:57:22,507 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2711 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2681 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:22,509 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2711 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2681 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:23,098 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2857 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:23,099 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2857 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:24,427 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:24,428 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2854 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:25,078 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 37.56 2021-05-03 01:57:25,746 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:25,746 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3083 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3155 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3113 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:28,160 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:28,161 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3230 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:29,786 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 13 docs from 0 to 5000 2021-05-03 01:57:30,211 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 40.09 2021-05-03 01:57:30,232 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3480 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3510 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3543 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:30,233 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3480 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3543 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3510 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:31,072 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3544 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:31,072 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3544 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:33,411 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3779 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:33,411 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3779 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:34,124 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:34,125 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3768 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:35,355 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 42.43 2021-05-03 01:57:35,661 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3945 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:35,663 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3945 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3848 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3951 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:38,556 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4051 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:38,558 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4051 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:38,559 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4051 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4057 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:38,559 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4051 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4057 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:40,535 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 44.86 2021-05-03 01:57:43,066 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:43,068 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4435 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:43,073 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:43,075 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4435 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:57:45,690 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 47.38 2021-05-03 01:57:50,812 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 50.02 2021-05-03 01:57:55,993 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 52.62 2021-05-03 01:57:57,201 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 0 to 5000 2021-05-03 01:57:57,203 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-03 01:57:57,203 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-03 01:57:57,203 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 0 to 5000 2021-05-03 01:57:57,206 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 8 docs from 0 to 5000 2021-05-03 01:57:57,206 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 0 to 5000 2021-05-03 01:57:57,207 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 01:57:59,927 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 01:58:01,196 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 55.38 2021-05-03 01:58:06,381 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 58.05 2021-05-03 01:58:11,596 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.85 2021-05-03 01:58:16,760 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 63.85 2021-05-03 01:58:21,953 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 01:58:27,197 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 01:58:32,519 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 01:58:37,489 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:EzWBogngS9TnfK1FZ-59-54000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:37,805 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:ujfeX_Ii-8Syud_0-59-55000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-253 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:38,154 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:g-3-59-55000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-253 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:38,756 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.85 2021-05-03 01:58:41,132 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-39 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-160 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:41,223 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-612 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-654 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-521 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:41,240 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-39 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-100 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-93 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-120 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-162 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-160 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:41,306 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-612 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-610 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-652 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-521 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-672 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:41,756 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:5rPLB7TIWKa-59-53000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:41,825 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:5rPLB7TIWKa-59-53000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:41,990 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-612 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-699 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-652 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:42,286 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:ujfeX_Ii-8Syud_0-59-55000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-469 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:42,290 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:g-3-59-55000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-492 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:42,338 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-79 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-120 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:42,404 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:6chhbz8kzHjruHI6-59-52000:ujfeX_Ii-8Syud_0-59-55000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-912 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-970 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:42,756 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:h2lpKyuICit5m-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-127 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-167 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-120 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:43,209 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-615 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-600 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-652 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:43,226 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:h2lpKyuICit5m-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-615 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-655 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-652 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:43,641 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-167 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-120 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,335 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 71.83 2021-05-03 01:58:44,447 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2845 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,448 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2854 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2845 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,519 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-318 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-247 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-256 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-202 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-288 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-392 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,519 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-318 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-247 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-256 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-202 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-288 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-392 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,698 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,700 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2580 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,749 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2771 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,750 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2854 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2771 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,971 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:44,979 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,016 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2730 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,017 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2730 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,188 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2362 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,190 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2362 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,334 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-328 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-318 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-288 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,369 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:cH2-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-134 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-151 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-174 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-171 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,378 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:lUJkz9a_jCjM-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-137 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-134 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-151 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-174 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-171 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,404 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:lUJkz9a_jCjM-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-606 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-603 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-623 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-645 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-689 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-547 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-646 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-520 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,424 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:cH2-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-606 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-603 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-623 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-689 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-547 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-646 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-520 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,834 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-787 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-793 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:46,926 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:h2lpKyuICit5m-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-318 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-288 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-251 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:47,273 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-796 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-793 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:47,285 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:h2lpKyuICit5m-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-796 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-793 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:47,562 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:h2lpKyuICit5m-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2743 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2898 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:47,562 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:h2lpKyuICit5m-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2743 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2777 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2898 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:47,936 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2908 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2898 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:47,937 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2908 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2898 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:48,335 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-448 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:48,339 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-448 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:48,671 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2898 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:48,671 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2898 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2777 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:48,770 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:7eNQIu1eCLa2MgtNR-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-975 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:48,786 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-975 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,101 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2816 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2766 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,101 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2766 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2774 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,401 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-643 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-577 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,437 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-28 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-131 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-173 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,516 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 74.46 2021-05-03 01:58:49,555 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2405 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2595 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,558 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2508 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2405 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2595 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2498 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,828 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:lUJkz9a_jCjM-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-395 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-242 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,832 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:cH2-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-395 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-242 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,915 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:lUJkz9a_jCjM-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-858 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-725 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:49,933 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:cH2-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-858 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-725 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:50,055 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-338 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-358 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-395 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:50,072 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:h2lpKyuICit5m-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-977 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:50,078 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:50,272 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:DLVvOgIn%D%evxVa-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-977 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,187 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,187 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,460 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-448 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,683 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,684 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,687 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-22 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,752 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2099 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:51,753 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2109 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2099 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:52,551 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2727 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:52,552 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2727 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:52,582 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-616 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-532 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-572 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:52,762 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-786 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:52,937 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-22 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:52,957 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:lUJkz9a_jCjM-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-927 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:52,963 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:cH2-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-927 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:53,303 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-245 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-265 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-200 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-231 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-220 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-240 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-390 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:53,401 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2528 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:53,401 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2528 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:53,451 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-537 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-653 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-552 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:53,651 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2583 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:53,653 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2583 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:54,726 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 77.14 2021-05-03 01:58:55,051 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-54 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-128 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-105 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,119 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-359 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-276 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,168 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,171 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,200 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-537 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-504 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-533 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-677 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,211 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-504 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-533 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-677 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,562 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-54 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-128 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-105 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,680 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2733 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:55,681 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2721 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2733 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:56,247 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:56,249 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:56,562 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2943 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2843 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:56,562 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2943 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2843 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:56,569 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-904 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-914 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-941 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:56,769 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-739 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-728 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-823 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-894 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-884 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:57,364 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-433 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:57,407 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-223 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-276 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:57,467 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-551 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:57,595 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2764 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:57,596 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:OJ-59-11000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2764 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2767 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:57,664 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-739 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-728 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-823 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-792 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,328 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2658 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2553 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,329 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2658 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2553 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,331 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-48 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-59 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-74 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,332 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-48 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-59 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-74 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,566 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-512 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-541 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-695 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,694 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,697 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2847 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,854 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-476 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-485 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-482 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,872 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-728 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-705 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-886 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-895 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-892 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-794 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:58,953 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-319 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-304 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-289 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-267 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-233 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-222 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-294 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-260 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:59,190 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2364 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2279 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:59,190 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2364 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2279 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:58:59,867 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.61 2021-05-03 01:59:00,420 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:00,421 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3078 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:00,641 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:34jjvMAR-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-916 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-966 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:00,812 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-456 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:00,901 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-339 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-267 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-233 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-222 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-260 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:00,940 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-87 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-156 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,040 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-795 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-740 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,048 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-605 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-557 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-624 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-622 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,575 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-208 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-398 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-274 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,592 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-208 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-398 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-274 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,673 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2706 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2742 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,674 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2706 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2742 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,690 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2706 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2742 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,691 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2706 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2742 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,757 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-728 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-705 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-886 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-895 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-892 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-794 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,846 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:01,848 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2705 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:02,677 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-408 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-449 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-456 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-445 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:02,796 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:1YsVPgym-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-968 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-954 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:03,151 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:03,153 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2529 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,523 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2765 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2823 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,525 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2765 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2823 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,562 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:XJ-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-947 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-966 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-964 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,576 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,609 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,611 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3655 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,661 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-919 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-917 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-968 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-954 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,964 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-236 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-312 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-282 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:04,979 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-497 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-460 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:05,066 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-839 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-713 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-889 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-887 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-799 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-756 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:05,071 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:G25hghz6kCnnWmIW-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-507 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-544 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-683 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-661 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-640 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:05,117 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 82.17 2021-05-03 01:59:05,832 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:G25hghz6kCnnWmIW-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-27 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-113 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-60 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:06,414 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:4VH5KF7dYn20KsMmxG-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-911 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-954 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:06,496 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:CosTpSvhTCIhs-JTVP-59-8000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-460 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-491 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:06,503 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-460 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-491 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:06,571 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:G25hghz6kCnnWmIW-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:06,572 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:G25hghz6kCnnWmIW-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2671 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:06,596 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2549 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:06,596 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:07,213 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-544 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-690 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:07,213 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-27 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-107 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,023 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2572 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2685 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,023 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2622 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2572 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2685 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,513 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,651 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-648 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-690 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,667 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-34 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-57 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,733 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2500 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,733 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2500 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2646 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,812 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-690 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:08,815 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:STVsE-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:09,042 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:09,111 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:09,111 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:09,687 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:7F7X-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-421 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:09,710 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:_default:G25hghz6kCnnWmIW-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-802 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-700 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-745 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:10,369 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 84.62 2021-05-03 01:59:10,838 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-34 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:10,898 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-627 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:10,915 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:10,987 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-106 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:11,381 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:11,382 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3373 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,298 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-379 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,421 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-379 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,424 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-379 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,503 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,555 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,555 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3817 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3987 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,763 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:G25hghz6kCnnWmIW-59-10000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-459 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,818 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:q0ZJrEp-59-9000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-77 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:12,845 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:q0ZJrEp-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-568 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,709 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-263 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,726 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-263 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,732 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,736 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-848 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-788 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,805 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,805 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2818 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,809 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,818 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,819 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:13,819 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2818 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:15,124 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-400 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:15,497 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-400 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:15,506 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-400 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:15,548 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 86.94 2021-05-03 01:59:15,947 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:q0ZJrEp-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-372 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:16,055 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:q0ZJrEp-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-810 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-743 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:16,105 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-590 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:16,947 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:4O%iF%s-59-7000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:16,960 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:16,960 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:18,119 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:w%u-59-11000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:18,125 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:18,647 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-36 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:18,726 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2512 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:18,726 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2512 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2627 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:20,638 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:KUZ%P1fVZ%xj-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:20,653 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:eipH-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-146 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:20,828 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 89.13 2021-05-03 01:59:20,957 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:q0ZJrEp-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-430 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:21,240 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:eipH-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-671 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:21,780 | test | INFO | pool-5-thread-9 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:22,375 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3439 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:22,377 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3439 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,042 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:TZYQVokZ7wPcamoYYwx-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-178 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,042 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:NI-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-178 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,513 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-311 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-281 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,585 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,586 | test | INFO | pool-5-thread-24 | [table_view:display:72] Keys failed in default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2605 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2517 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,594 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:NI-59-7000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,595 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:_default:NI-59-7000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2503 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2517 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2605 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,602 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,604 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2863 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,664 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-61 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-111 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-178 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-82 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,732 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-663 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-680 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,755 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3806 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3996 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:23,756 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3806 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3996 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:24,079 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:_default:eipH-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-322 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-378 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:24,161 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:YzrzWdVpYCX_fqNzW-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-579 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-680 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:24,220 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:YzrzWdVpYCX_fqNzW-59-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:24,220 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:YzrzWdVpYCX_fqNzW-59-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:24,292 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:nJPk3GLaB751nVIcOqP4-59-6000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-24 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:24,339 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:nJPk3GLaB751nVIcOqP4-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-680 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:24,792 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:eipH-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-825 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-755 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-710 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:25,414 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3321 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:25,415 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3321 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:26,030 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3756 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:26,032 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3756 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:26,055 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 91.54 2021-05-03 01:59:26,184 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:TZYQVokZ7wPcamoYYwx-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:26,187 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:TZYQVokZ7wPcamoYYwx-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2897 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:26,490 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:TZYQVokZ7wPcamoYYwx-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-867 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:26,821 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:_default:cxthWt8E4gz2T8yu-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-409 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:27,220 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:NI-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-867 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:27,421 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-305 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-295 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:27,536 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-789 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:28,184 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:nJPk3GLaB751nVIcOqP4-59-6000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-789 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:28,187 | test | INFO | pool-5-thread-21 | [table_view:display:72] Keys failed in default:_default:YzrzWdVpYCX_fqNzW-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-789 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:28,188 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:nJPk3GLaB751nVIcOqP4-59-6000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2890 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:28,190 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:nJPk3GLaB751nVIcOqP4-59-6000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2890 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:28,197 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:YzrzWdVpYCX_fqNzW-59-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2890 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:28,197 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:YzrzWdVpYCX_fqNzW-59-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2890 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:29,055 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:29,056 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3523 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:30,137 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:N_PXW2o-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:30,138 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:_default:N_PXW2o-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:31,243 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 94.07 2021-05-03 01:59:33,209 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:W-59-9000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-510 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:35,970 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:seIWC-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-586 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:36,526 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 96.84 2021-05-03 01:59:37,365 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:F9s139WOKc2hKwtSJ4Hl5-59-11000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:37,367 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:_default:F9s139WOKc2hKwtSJ4Hl5-59-11000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:39,265 | test | INFO | pool-5-thread-27 | [table_view:display:72] Keys failed in default:_default:W-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-714 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-711 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-741 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:41,595 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:_default:6rbId54Fe-tLPHl4vN%Y6K-59-7000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-139 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:41,683 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 99.66 2021-05-03 01:59:41,743 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:_default:seIWC-59-10000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-775 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,051 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:seIWC-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,052 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:seIWC-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2829 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,770 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:_default:Lw%4hU7krM7DqoKZwt_-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-188 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,855 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:Lw%4hU7krM7DqoKZwt_-59-8000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,857 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:_default:Lw%4hU7krM7DqoKZwt_-59-8000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,875 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:Lw%4hU7krM7DqoKZwt_-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-559 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,924 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in default:_default:Q2-j4REGBP7ZomoQFwY-59-8000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-44 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,961 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:W-59-9000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-405 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-424 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:42,963 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:Q2-j4REGBP7ZomoQFwY-59-8000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-629 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-608 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:43,532 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4423 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:43,535 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4423 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:43,612 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 01:59:43,983 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:D-VsvsD-59-10000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:43,983 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:_default:D-VsvsD-59-10000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2541 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 01:59:46,786 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 01:59:47,582 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-03 01:59:51,398 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-03 01:59:51,914 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:00:01,144 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-03 02:00:01,154 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 02:00:01,154 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 02:00:01,154 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 02:00:01,155 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-03 02:00:01,157 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 02:00:01,157 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 02:00:01,158 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-03 02:00:01,158 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 11 docs from 0 to 500 2021-05-03 02:00:01,160 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,161 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 02:00:01,161 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 0 to 500 2021-05-03 02:00:01,161 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,163 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 15 docs from 500 to 980 2021-05-03 02:00:01,163 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-03 02:00:01,164 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,164 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 500 to 980 2021-05-03 02:00:01,164 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 13 docs from 0 to 500 2021-05-03 02:00:01,165 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 02:00:01,165 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 500 to 980 2021-05-03 02:00:01,167 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 0 to 500 2021-05-03 02:00:01,167 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 2500 to 2900 2021-05-03 02:00:01,167 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 500 to 980 2021-05-03 02:00:01,168 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 10 docs from 0 to 500 2021-05-03 02:00:01,170 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 13 docs from 500 to 980 2021-05-03 02:00:01,170 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-03 02:00:01,171 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-03 02:00:01,171 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 13 docs from 0 to 500 2021-05-03 02:00:01,171 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 02:00:01,173 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 500 to 980 2021-05-03 02:00:01,174 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-03 02:00:01,174 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 02:00:01,176 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 14 docs from 500 to 980 2021-05-03 02:00:01,176 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 15 docs from 0 to 500 2021-05-03 02:00:01,177 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,177 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 15 docs from 500 to 980 2021-05-03 02:00:01,177 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-03 02:00:01,177 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,178 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 15 docs from 500 to 980 2021-05-03 02:00:01,180 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 20 docs from 0 to 500 2021-05-03 02:00:01,180 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,180 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-03 02:00:01,181 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-03 02:00:01,181 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 2500 to 2900 2021-05-03 02:00:01,183 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 02:00:01,184 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-03 02:00:01,184 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,186 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 13 docs from 500 to 980 2021-05-03 02:00:01,186 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 10 docs from 0 to 500 2021-05-03 02:00:01,187 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,187 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 02:00:01,188 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-03 02:00:01,188 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 02:00:01,190 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 02:00:01,190 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 6 docs from 0 to 500 2021-05-03 02:00:01,191 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,193 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 02:00:01,194 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-03 02:00:01,194 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,196 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-03 02:00:01,196 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-03 02:00:01,197 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,197 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 02:00:01,197 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 0 to 500 2021-05-03 02:00:01,198 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,198 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-03 02:00:01,200 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 0 to 500 2021-05-03 02:00:01,200 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,200 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-03 02:00:01,201 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 0 to 500 2021-05-03 02:00:01,201 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 6 docs from 500 to 980 2021-05-03 02:00:01,203 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 02:00:01,203 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 02:00:01,203 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 6 docs from 0 to 500 2021-05-03 02:00:01,204 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 02:00:01,206 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 02:00:01,206 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 02:00:01,207 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,207 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-03 02:00:01,207 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-03 02:00:01,209 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,210 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 02:00:01,210 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 02:00:01,210 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 02:00:01,211 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 02:00:01,211 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 02:00:01,213 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 02:00:01,213 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 02:00:01,213 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,214 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,216 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-03 02:00:01,217 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 02:00:01,217 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 02:00:01,217 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,219 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,219 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 02:00:01,220 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 02:00:01,220 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:01,220 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 02:00:01,221 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 02:00:01,223 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 02:00:01,223 | test | ERROR | pool-5-thread-30 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 02:00:02,055 | test | INFO | pool-5-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1176e0 type=client open_count=1 channel=[id: 0xf742197d, 0.0.0.0/0.0.0.0:34195] timeout=300.0>) 2021-05-03 02:00:02,105 | test | ERROR | pool-5-thread-1 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1176ff type=client open_count=1 channel=[id: 0x45f69f7d, 0.0.0.0/0.0.0.0:48131] timeout=300.0>) 2021-05-03 02:00:04,137 | test | ERROR | pool-5-thread-1 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:00:06,302 | test | INFO | pool-5-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 254.123000145 sec 2021-05-03 02:01:25,887 | test | INFO | pool-5-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.*...*......*...**..................**..........*....*.......*...**.....X | 432.646 | | 417.025 | .........................................................**..*........*........X | 542.070 | | 536.269 | ..........................................................................*...................*......X | 712.700 | | 702.600 | .................................................................................................*X | 702.600 | | 539.600 | ..........................................................................X | 539.600 | | 381.677 | ....................................................X | 381.677 | | 230.677 | ...............................X | 230.677 | | 227.477 | ..............................**....*.........X | 318.930 | | 298.430 | ........................................*..X | 309.600 | | 284.200 | ......................................*.......*X | 334.900 | | 327.700 | ............................................X | 327.700 | | 322.300 | ............................................X | 322.300 | | 238.400 | ................................*X | 238.400 | | 55.200 | ......*X | 55.200 | | 5.700 | *X | 5.700 | | 0.000 | ********************X | 5.399 | | 3.599 | *X | 3.599 | | 1.800 | X | 1.800 | | 0.000 | ************************.*....**........**...........*.......**......X | 265.200 | | 252.333 | ..................................X | 252.333 | | 217.733 | .............................*X | 217.733 | | 160.633 | .....................*X | 160.633 | | 94.908 | ............X | 94.908 | | 53.308 | ......*X | 53.308 | | 22.108 | ..*X | 22.108 | | 14.900 | .**X | 14.900 | | 12.500 | X | 12.500 | | 9.200 | X | 9.200 | | 3.500 | *X | 3.500 | | 2.300 | *X | 2.300 | | 0.800 | *X | 7.100 | +---------+--------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:01:27,815 | test | INFO | pool-5-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **........**.......*........*.......X | 875.249 | | 816.049 | ...........................*X | 816.049 | | 427.985 | .............*X | 427.985 | | 410.385 | .............*..*.....X | 613.583 | | 567.383 | ..................*X | 567.383 | | 271.243 | ........*X | 271.243 | | 225.743 | ......**.....**...X | 463.873 | | 435.410 | .............*.*X | 463.526 | | 420.126 | .............*.*.*..**.**X | 577.478 | | 538.578 | .................X | 538.578 | | 526.434 | .................*...*..**.****..*X | 778.108 | | 750.508 | ........................*.X | 759.108 | | 695.287 | ......................*.X | 721.528 | | 721.528 | .......................*...*....*X | 924.079 | | 823.610 | ...........................X | 823.610 | | 781.010 | .........................*...**............**..*..X | 1331.656 | | 1271.486 | ..........................................X | 1271.486 | | 1229.386 | .........................................*X | 1229.386 | | 1141.809 | ......................................*.*.*X | 1219.315 | | 1095.866 | ....................................*X | 1095.866 | | 1050.367 | ...................................*..*..*.*X | 1202.967 | | 1166.391 | .......................................X | 1166.391 | | 1041.138 | ..................................*...**.............*................**...............*..X | 2456.827 | | 2396.261 | .................................................................................*X | 2396.261 | | 2338.684 | ...............................................................................**.......*......**.......X | 2908.800 | | 2893.700 | ..................................................................................................X | 2893.700 | | 2842.100 | ................................................................................................*X | 2842.100 | | 2587.700 | .......................................................................................*X | 2587.700 | | 2260.000 | ............................................................................X | 2260.000 | | 1975.000 | ..................................................................X | 1975.000 | | 1880.700 | ...............................................................**.X | 1895.827 | | 1846.127 | ..............................................................*X | 1846.127 | | 1639.627 | .......................................................*X | 1639.627 | | 1232.206 | .........................................X | 1232.206 | | 766.506 | .........................X | 766.506 | | 723.906 | .......................**.............*.........................**........**X | 2048.700 | | 1756.900 | ...........................................................*.**.....*............*X | 2278.700 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 02:01:28,407 | test | INFO | pool-5-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *********.......*....**.*...X | 102.099 | | 92.299 | .............**.**..***.....*...*.X | 173.505 | | 160.307 | .......................*X | 160.307 | | 136.707 | ....................X | 136.707 | | 131.698 | ...................X | 131.698 | | 131.490 | ...................*...**..*..*.*X | 183.896 | | 131.207 | ...................*X | 131.207 | | 92.107 | .............X | 92.107 | | 55.707 | .......X | 55.707 | | 12.800 | X | 12.800 | | 0.000 | **************************************************************..*..........**..................**......................................................***................X | 649.987 | | 631.100 | ................................................................................................X | 631.100 | | 566.000 | ......................................................................................*X | 566.000 | | 484.700 | .........................................................................*X | 484.700 | | 447.900 | ...................................................................*..**.....*X | 489.600 | | 411.800 | ..............................................................*X | 411.800 | | 309.700 | ..............................................X | 309.700 | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:01:29,056 | test | WARNING | Thread-8186 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aDSQxlcxTTTSiyZHm%-1-169000:Obd%enza-4-678000 create failed keys from task: [u'test_collections-1639', u'test_collections-426', u'test_collections-1549', u'test_collections-192', u'test_collections-1321'] 2021-05-03 02:01:29,059 | test | WARNING | Thread-8187 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aDSQxlcxTTTSiyZHm%-1-169000:IdvY-1-397000 create failed keys from task: [u'test_collections-1639', u'test_collections-426', u'test_collections-1549', u'test_collections-192', u'test_collections-1321'] 2021-05-03 02:01:29,061 | test | WARNING | Thread-8188 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aDSQxlcxTTTSiyZHm%-1-169000:NPwdTe4-wIU-1-454000 create failed keys from task: [u'test_collections-1639', u'test_collections-426', u'test_collections-1549', u'test_collections-192'] 2021-05-03 02:01:29,061 | test | WARNING | Thread-8189 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:6ADiXxcxF2QZ_0H-1-771000:t5jzAWJRmG_s6-2-52000 create failed keys from task: [u'test_collections-528'] 2021-05-03 02:01:29,062 | test | WARNING | Thread-8190 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:6ADiXxcxF2QZ_0H-1-771000:v2oMI3Sg_P35%5lnm-2-29000 create failed keys from task: [u'test_collections-528', u'test_collections-33'] 2021-05-03 02:01:29,082 | test | WARNING | Thread-8191 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:0QSIhiJqxmtqWa-2-429000:MSID5iMr-2-572000 create failed keys from task: [u'test_collections-729', u'test_collections-128', u'test_collections-667', u'test_collections-327', u'test_collections-540', u'test_collections-2155', u'test_collections-1288', u'test_collections-2092', u'test_collections-339', u'test_collections-1318', u'test_collections-1570', u'test_collections-1771', u'test_collections-2102'] 2021-05-03 02:01:29,148 | test | WARNING | Thread-8192 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:0QSIhiJqxmtqWa-2-429000:5-2-573000 create failed keys from task: [u'test_collections-1639', u'test_collections-426', u'test_collections-1549', u'test_collections-192'] 2021-05-03 02:01:29,227 | test | WARNING | Thread-8270 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7eNQIu1eCLa2MgtNR-59-10000 update failed keys from task: [u'test_collections-829', u'test_collections-868', u'test_collections-967', u'test_collections-612', u'test_collections-975', u'test_collections-654', u'test_collections-784', u'test_collections-521'] 2021-05-03 02:01:29,230 | test | WARNING | Thread-8271 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 update failed keys from task: [u'test_collections-829', u'test_collections-868', u'test_collections-967', u'test_collections-612', u'test_collections-975', u'test_collections-610', u'test_collections-652', u'test_collections-784', u'test_collections-521', u'test_collections-672'] 2021-05-03 02:01:29,230 | test | WARNING | Thread-8273 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:h2lpKyuICit5m-59-8000 update failed keys from task: [u'test_collections-907', u'test_collections-926', u'test_collections-922', u'test_collections-524', u'test_collections-920', u'test_collections-781', u'test_collections-977', u'test_collections-615', u'test_collections-953', u'test_collections-698', u'test_collections-655', u'test_collections-652', u'test_collections-796', u'test_collections-793'] 2021-05-03 02:01:29,233 | test | WARNING | Thread-8274 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DLVvOgIn%D%evxVa-59-8000 update failed keys from task: [u'test_collections-907', u'test_collections-926', u'test_collections-922', u'test_collections-524', u'test_collections-920', u'test_collections-600', u'test_collections-781', u'test_collections-977', u'test_collections-615', u'test_collections-953', u'test_collections-698', u'test_collections-652', u'test_collections-796', u'test_collections-793'] 2021-05-03 02:01:29,233 | test | WARNING | Thread-8275 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:lUJkz9a_jCjM-59-6000 update failed keys from task: [u'test_collections-927', u'test_collections-606', u'test_collections-603', u'test_collections-725', u'test_collections-623', u'test_collections-645', u'test_collections-689', u'test_collections-942', u'test_collections-547', u'test_collections-646', u'test_collections-566', u'test_collections-520', u'test_collections-858', u'test_collections-791'] 2021-05-03 02:01:29,234 | test | WARNING | Thread-8276 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:cH2-59-10000 update failed keys from task: [u'test_collections-927', u'test_collections-606', u'test_collections-603', u'test_collections-725', u'test_collections-623', u'test_collections-689', u'test_collections-942', u'test_collections-547', u'test_collections-646', u'test_collections-566', u'test_collections-520', u'test_collections-858', u'test_collections-791'] 2021-05-03 02:01:29,236 | test | WARNING | Thread-8277 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:OJ-59-11000 update failed keys from task: [u'test_collections-904', u'test_collections-914', u'test_collections-643', u'test_collections-577', u'test_collections-621', u'test_collections-786', u'test_collections-941'] 2021-05-03 02:01:29,236 | test | WARNING | Thread-8278 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:34jjvMAR-59-8000 update failed keys from task: [u'test_collections-728', u'test_collections-823', u'test_collections-966', u'test_collections-641', u'test_collections-543', u'test_collections-884', u'test_collections-939', u'test_collections-739', u'test_collections-916', u'test_collections-517', u'test_collections-616', u'test_collections-894', u'test_collections-532', u'test_collections-572'] 2021-05-03 02:01:29,230 | test | WARNING | Thread-8272 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 update failed keys from task: [u'test_collections-907', u'test_collections-926', u'test_collections-749', u'test_collections-922', u'test_collections-967', u'test_collections-524', u'test_collections-920', u'test_collections-787', u'test_collections-561', u'test_collections-977', u'test_collections-612', u'test_collections-953', u'test_collections-699', u'test_collections-652', u'test_collections-793'] 2021-05-03 02:01:29,236 | test | WARNING | Thread-8279 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4VH5KF7dYn20KsMmxG-59-8000 update failed keys from task: [u'test_collections-728', u'test_collections-705', u'test_collections-504', u'test_collections-866', u'test_collections-943', u'test_collections-886', u'test_collections-641', u'test_collections-911', u'test_collections-954', u'test_collections-533', u'test_collections-677', u'test_collections-895', u'test_collections-892', u'test_collections-794'] 2021-05-03 02:01:31,950 | test | WARNING | Thread-8277 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:OJ-59-11000 delete failed keys from task: [u'test_collections-245', u'test_collections-265', u'test_collections-200', u'test_collections-220', u'test_collections-240', u'test_collections-28', u'test_collections-116', u'test_collections-433', u'test_collections-131', u'test_collections-153', u'test_collections-231', u'test_collections-173', u'test_collections-390'] 2021-05-03 02:01:32,020 | test | WARNING | Thread-8275 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:lUJkz9a_jCjM-59-6000 delete failed keys from task: [u'test_collections-205', u'test_collections-137', u'test_collections-215', u'test_collections-134', u'test_collections-395', u'test_collections-385', u'test_collections-242', u'test_collections-151', u'test_collections-174', u'test_collections-171'] 2021-05-03 02:01:32,022 | test | WARNING | Thread-8270 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7eNQIu1eCLa2MgtNR-59-10000 delete failed keys from task: [u'test_collections-39', u'test_collections-448', u'test_collections-126', u'test_collections-247', u'test_collections-202', u'test_collections-288', u'test_collections-160', u'test_collections-318', u'test_collections-256', u'test_collections-235', u'test_collections-392'] 2021-05-03 02:01:32,042 | test | WARNING | Thread-8276 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:cH2-59-10000 delete failed keys from task: [u'test_collections-205', u'test_collections-215', u'test_collections-134', u'test_collections-395', u'test_collections-385', u'test_collections-242', u'test_collections-151', u'test_collections-174', u'test_collections-171'] 2021-05-03 02:01:32,075 | test | WARNING | Thread-8271 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 delete failed keys from task: [u'test_collections-39', u'test_collections-448', u'test_collections-247', u'test_collections-202', u'test_collections-100', u'test_collections-288', u'test_collections-120', u'test_collections-162', u'test_collections-160', u'test_collections-318', u'test_collections-256', u'test_collections-235', u'test_collections-93', u'test_collections-392'] 2021-05-03 02:01:32,081 | test | WARNING | Thread-8279 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4VH5KF7dYn20KsMmxG-59-8000 delete failed keys from task: [u'test_collections-54', u'test_collections-128', u'test_collections-96', u'test_collections-225', u'test_collections-105', u'test_collections-204', u'test_collections-267', u'test_collections-222', u'test_collections-460', u'test_collections-260', u'test_collections-339', u'test_collections-133', u'test_collections-233', u'test_collections-497', u'test_collections-394'] 2021-05-03 02:01:32,082 | test | WARNING | Thread-8274 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DLVvOgIn%D%evxVa-59-8000 delete failed keys from task: [u'test_collections-12', u'test_collections-205', u'test_collections-167', u'test_collections-120', u'test_collections-384', u'test_collections-182', u'test_collections-4', u'test_collections-338', u'test_collections-214', u'test_collections-358', u'test_collections-252', u'test_collections-395', u'test_collections-172', u'test_collections-6'] 2021-05-03 02:01:32,107 | test | WARNING | Thread-8278 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:34jjvMAR-59-8000 delete failed keys from task: [u'test_collections-22', u'test_collections-359', u'test_collections-476', u'test_collections-133', u'test_collections-276', u'test_collections-485', u'test_collections-164', u'test_collections-241', u'test_collections-482'] 2021-05-03 02:01:32,115 | test | WARNING | Thread-8273 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:h2lpKyuICit5m-59-8000 delete failed keys from task: [u'test_collections-78', u'test_collections-127', u'test_collections-167', u'test_collections-288', u'test_collections-120', u'test_collections-384', u'test_collections-182', u'test_collections-4', u'test_collections-318', u'test_collections-214', u'test_collections-252', u'test_collections-251', u'test_collections-6'] 2021-05-03 02:01:32,121 | test | WARNING | Thread-8272 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 delete failed keys from task: [u'test_collections-328', u'test_collections-79', u'test_collections-448', u'test_collections-288', u'test_collections-120', u'test_collections-384', u'test_collections-2', u'test_collections-4', u'test_collections-318', u'test_collections-214', u'test_collections-235', u'test_collections-252'] 2021-05-03 02:01:32,121 | test | WARNING | Thread-8270 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7eNQIu1eCLa2MgtNR-59-10000 create failed keys from task: [u'test_collections-2854', u'test_collections-2845'] 2021-05-03 02:01:32,140 | test | WARNING | Thread-8277 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:OJ-59-11000 create failed keys from task: [u'test_collections-2764', u'test_collections-2767', u'test_collections-2528'] 2021-05-03 02:01:32,154 | test | WARNING | Thread-8271 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Fs392lIpJiOgsc0Xn09_-59-9000 create failed keys from task: [u'test_collections-2854', u'test_collections-2771'] 2021-05-03 02:01:32,164 | test | WARNING | Thread-8278 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:34jjvMAR-59-8000 create failed keys from task: [u'test_collections-2721', u'test_collections-2733', u'test_collections-2582'] 2021-05-03 02:01:32,181 | test | WARNING | Thread-8274 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DLVvOgIn%D%evxVa-59-8000 create failed keys from task: [u'test_collections-2816', u'test_collections-2766', u'test_collections-2580', u'test_collections-2774'] 2021-05-03 02:01:32,206 | test | WARNING | Thread-8279 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4VH5KF7dYn20KsMmxG-59-8000 create failed keys from task: [u'test_collections-2548', u'test_collections-2705'] 2021-05-03 02:01:32,209 | test | WARNING | Thread-8273 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:h2lpKyuICit5m-59-8000 create failed keys from task: [u'test_collections-2743', u'test_collections-2777', u'test_collections-2898'] 2021-05-03 02:01:32,209 | test | WARNING | Thread-8272 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:6U1d8%Et0iruFBCT-Gmk-59-10000 create failed keys from task: [u'test_collections-2898', u'test_collections-2777'] 2021-05-03 02:01:32,240 | test | WARNING | Thread-8647 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:F44ziveBz7bZyb7PEugb-57-475000:4vPV6U5mYu4Ktu%-57-663000 create failed keys from task: [u'test_collections-4051', u'test_collections-3768', u'test_collections-2711', u'test_collections-2854', u'test_collections-4057', u'test_collections-2681', u'test_collections-4435', u'test_collections-3544'] 2021-05-03 02:01:32,249 | test | WARNING | Thread-8648 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:F44ziveBz7bZyb7PEugb-57-475000:aybXD%exjV-57-785000 create failed keys from task: [u'test_collections-2857', u'test_collections-3945', u'test_collections-3848', u'test_collections-4435', u'test_collections-3543', u'test_collections-3083', u'test_collections-3480', u'test_collections-4051', u'test_collections-3779', u'test_collections-3230', u'test_collections-3155', u'test_collections-3113', u'test_collections-3951', u'test_collections-3510'] 2021-05-03 02:01:32,280 | test | WARNING | Thread-8283 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:CosTpSvhTCIhs-JTVP-59-8000 update failed keys from task: [u'test_collections-607', u'test_collections-968', u'test_collections-846', u'test_collections-602', u'test_collections-943', u'test_collections-740', u'test_collections-919', u'test_collections-917', u'test_collections-954', u'test_collections-795', u'test_collections-551'] 2021-05-03 02:01:32,417 | test | WARNING | Thread-8283 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:CosTpSvhTCIhs-JTVP-59-8000 delete failed keys from task: [u'test_collections-59', u'test_collections-208', u'test_collections-74', u'test_collections-225', u'test_collections-324', u'test_collections-460', u'test_collections-48', u'test_collections-49', u'test_collections-398', u'test_collections-274', u'test_collections-471', u'test_collections-491'] 2021-05-03 02:01:32,522 | test | WARNING | Thread-8283 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:CosTpSvhTCIhs-JTVP-59-8000 create failed keys from task: [u'test_collections-2706', u'test_collections-2742'] 2021-05-03 02:01:33,904 | test | WARNING | Thread-8286 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:G25hghz6kCnnWmIW-59-10000 update failed keys from task: [u'test_collections-609', u'test_collections-707', u'test_collections-507', u'test_collections-802', u'test_collections-700', u'test_collections-745', u'test_collections-544', u'test_collections-683', u'test_collections-661', u'test_collections-640'] 2021-05-03 02:01:33,905 | test | WARNING | Thread-8288 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default update failed keys from task: [u'test_collections-609', u'test_collections-876', u'test_collections-697', u'test_collections-690'] 2021-05-03 02:01:33,907 | test | WARNING | Thread-8280 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:XJ-59-7000 update failed keys from task: [u'test_collections-728', u'test_collections-947', u'test_collections-823', u'test_collections-966', u'test_collections-964', u'test_collections-866', u'test_collections-641', u'test_collections-540', u'test_collections-939', u'test_collections-739', u'test_collections-537', u'test_collections-797', u'test_collections-653', u'test_collections-552', u'test_collections-792'] 2021-05-03 02:01:33,907 | test | WARNING | Thread-8287 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4O%iF%s-59-7000 update failed keys from task: [u'test_collections-609', u'test_collections-848', u'test_collections-944', u'test_collections-516', u'test_collections-744', u'test_collections-788', u'test_collections-876', u'test_collections-544', u'test_collections-675', u'test_collections-690'] 2021-05-03 02:01:33,907 | test | WARNING | Thread-8284 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7F7X-59-9000 update failed keys from task: [u'test_collections-605', u'test_collections-506', u'test_collections-889', u'test_collections-887', u'test_collections-624', u'test_collections-622', u'test_collections-839', u'test_collections-979', u'test_collections-713', u'test_collections-557', u'test_collections-832', u'test_collections-799', u'test_collections-756'] 2021-05-03 02:01:33,908 | test | WARNING | Thread-8285 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:KUZ%P1fVZ%xj-59-6000 update failed keys from task: [u'test_collections-609', u'test_collections-944', u'test_collections-648', u'test_collections-876', u'test_collections-690'] 2021-05-03 02:01:33,910 | test | WARNING | Thread-8289 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 update failed keys from task: [u'test_collections-706', u'test_collections-627', u'test_collections-669', u'test_collections-944', u'test_collections-862'] 2021-05-03 02:01:33,911 | test | WARNING | Thread-8290 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:w%u-59-11000 update failed keys from task: [u'test_collections-706', u'test_collections-669', u'test_collections-944', u'test_collections-862'] 2021-05-03 02:01:33,911 | test | WARNING | Thread-8291 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:q0ZJrEp-59-9000 update failed keys from task: [u'test_collections-568', u'test_collections-810', u'test_collections-743', u'test_collections-773'] 2021-05-03 02:01:33,936 | test | WARNING | Thread-8303 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Lw%4hU7krM7DqoKZwt_-59-8000 update failed keys from task: [u'test_collections-559'] 2021-05-03 02:01:33,946 | test | WARNING | Thread-8291 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:q0ZJrEp-59-9000 delete failed keys from task: [u'test_collections-76', u'test_collections-77', u'test_collections-83', u'test_collections-430', u'test_collections-372'] 2021-05-03 02:01:33,959 | test | WARNING | Thread-8303 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Lw%4hU7krM7DqoKZwt_-59-8000 delete failed keys from task: [u'test_collections-42', u'test_collections-188'] 2021-05-03 02:01:33,969 | test | WARNING | Thread-8289 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 delete failed keys from task: [u'test_collections-34', u'test_collections-431', u'test_collections-263'] 2021-05-03 02:01:33,970 | test | WARNING | Thread-8290 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:w%u-59-11000 delete failed keys from task: [u'test_collections-106', u'test_collections-95', u'test_collections-155', u'test_collections-431', u'test_collections-263'] 2021-05-03 02:01:33,977 | test | WARNING | Thread-8285 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:KUZ%P1fVZ%xj-59-6000 delete failed keys from task: [u'test_collections-33', u'test_collections-34', u'test_collections-94', u'test_collections-379', u'test_collections-346', u'test_collections-400', u'test_collections-71'] 2021-05-03 02:01:33,979 | test | WARNING | Thread-8288 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default delete failed keys from task: [u'test_collections-34', u'test_collections-57', u'test_collections-109', u'test_collections-379', u'test_collections-346', u'test_collections-400', u'test_collections-71'] 2021-05-03 02:01:33,986 | test | WARNING | Thread-8287 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4O%iF%s-59-7000 delete failed keys from task: [u'test_collections-27', u'test_collections-33', u'test_collections-94', u'test_collections-107', u'test_collections-379', u'test_collections-344', u'test_collections-455', u'test_collections-400'] 2021-05-03 02:01:34,002 | test | WARNING | Thread-8289 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:T6Te43x7%Vk9d5ptv_Nfx-59-9000 create failed keys from task: [u'test_collections-2818'] 2021-05-03 02:01:34,033 | test | WARNING | Thread-8303 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Lw%4hU7krM7DqoKZwt_-59-8000 create failed keys from task: [u'test_collections-2653'] 2021-05-03 02:01:34,033 | test | WARNING | Thread-8290 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:w%u-59-11000 create failed keys from task: [u'test_collections-2818'] 2021-05-03 02:01:34,042 | test | WARNING | Thread-8287 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4O%iF%s-59-7000 create failed keys from task: [u'test_collections-2549'] 2021-05-03 02:01:34,042 | test | WARNING | Thread-8288 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default create failed keys from task: [u'test_collections-2500', u'test_collections-2646'] 2021-05-03 02:01:34,046 | test | WARNING | Thread-8286 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:G25hghz6kCnnWmIW-59-10000 delete failed keys from task: [u'test_collections-27', u'test_collections-11', u'test_collections-459', u'test_collections-113', u'test_collections-60', u'test_collections-132'] 2021-05-03 02:01:34,101 | test | WARNING | Thread-8284 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7F7X-59-9000 delete failed keys from task: [u'test_collections-87', u'test_collections-236', u'test_collections-31', u'test_collections-421', u'test_collections-312', u'test_collections-177', u'test_collections-156', u'test_collections-275', u'test_collections-150', u'test_collections-282'] 2021-05-03 02:01:34,121 | test | WARNING | Thread-8280 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:XJ-59-7000 delete failed keys from task: [u'test_collections-22', u'test_collections-456', u'test_collections-223', u'test_collections-133', u'test_collections-276', u'test_collections-266', u'test_collections-121'] 2021-05-03 02:01:34,121 | test | WARNING | Thread-8286 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:G25hghz6kCnnWmIW-59-10000 create failed keys from task: [u'test_collections-2671'] 2021-05-03 02:01:34,157 | test | WARNING | Thread-8285 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:KUZ%P1fVZ%xj-59-6000 create failed keys from task: [u'test_collections-2622', u'test_collections-2572', u'test_collections-2685'] 2021-05-03 02:01:34,211 | test | WARNING | Thread-8284 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7F7X-59-9000 create failed keys from task: [u'test_collections-2765', u'test_collections-2823'] 2021-05-03 02:01:34,217 | test | WARNING | Thread-8280 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:XJ-59-7000 create failed keys from task: [u'test_collections-2847', u'test_collections-2583'] 2021-05-03 02:01:34,253 | test | WARNING | Thread-8331 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:6chhbz8kzHjruHI6-59-52000:ujfeX_Ii-8Syud_0-59-55000 update failed keys from task: [u'test_collections-912', u'test_collections-934', u'test_collections-970'] 2021-05-03 02:01:34,290 | test | WARNING | Thread-8331 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:6chhbz8kzHjruHI6-59-52000:ujfeX_Ii-8Syud_0-59-55000 delete failed keys from task: [u'test_collections-216', u'test_collections-469', u'test_collections-386', u'test_collections-253'] 2021-05-03 02:01:35,561 | test | WARNING | Thread-8306 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Q2-j4REGBP7ZomoQFwY-59-8000 update failed keys from task: [u'test_collections-629', u'test_collections-608'] 2021-05-03 02:01:35,562 | test | WARNING | Thread-8304 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:seIWC-59-10000 update failed keys from task: [u'test_collections-775', u'test_collections-586'] 2021-05-03 02:01:35,575 | test | WARNING | Thread-8329 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:6chhbz8kzHjruHI6-59-52000:EzWBogngS9TnfK1FZ-59-54000 update failed keys from task: [u'test_collections-931', u'test_collections-974'] 2021-05-03 02:01:35,578 | test | WARNING | Thread-8330 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:6chhbz8kzHjruHI6-59-52000:5rPLB7TIWKa-59-53000 update failed keys from task: [u'test_collections-971'] 2021-05-03 02:01:35,578 | test | WARNING | Thread-8332 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:6chhbz8kzHjruHI6-59-52000:g-3-59-55000 delete failed keys from task: [u'test_collections-216', u'test_collections-386', u'test_collections-253', u'test_collections-492'] 2021-05-03 02:01:35,579 | test | WARNING | Thread-8306 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Q2-j4REGBP7ZomoQFwY-59-8000 delete failed keys from task: [u'test_collections-44'] 2021-05-03 02:01:35,581 | test | WARNING | Thread-8304 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:seIWC-59-10000 create failed keys from task: [u'test_collections-2829'] 2021-05-03 02:01:35,585 | test | WARNING | Thread-8349 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:kOcMKXakZMJ-3-571000:7gLUK75t-3-789000 create failed keys from task: [u'test_collections-2508', u'test_collections-3756', u'test_collections-2405', u'test_collections-3439', u'test_collections-2727', u'test_collections-3373', u'test_collections-2362', u'test_collections-2595', u'test_collections-2498'] 2021-05-03 02:01:35,589 | test | WARNING | Thread-8350 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:kOcMKXakZMJ-3-571000:pqESjOAtm-F%xepA-3-770000 create failed keys from task: [u'test_collections-3523', u'test_collections-2109', u'test_collections-2529', u'test_collections-2099', u'test_collections-2364', u'test_collections-3321', u'test_collections-2279'] 2021-05-03 02:01:35,601 | test | WARNING | Thread-8330 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:6chhbz8kzHjruHI6-59-52000:5rPLB7TIWKa-59-53000 delete failed keys from task: [u'test_collections-438'] 2021-05-03 02:01:35,614 | test | WARNING | Thread-8387 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:UoJpJyURnp6Ww-4-99000:YYTEaqv0kt7OextSG-4-303000 create failed keys from task: [u'test_collections-3817', u'test_collections-2908', u'test_collections-3655', u'test_collections-2898', u'test_collections-3987', u'test_collections-3146'] 2021-05-03 02:01:35,625 | test | WARNING | Thread-8388 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:UoJpJyURnp6Ww-4-99000:PmMfs-4-263000 create failed keys from task: [u'test_collections-3391', u'test_collections-3806', u'test_collections-3996', u'test_collections-2943', u'test_collections-2843', u'test_collections-3078', u'test_collections-3201', u'test_collections-4423', u'test_collections-2730'] 2021-05-03 02:01:36,102 | test | WARNING | Thread-8281 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:STVsE-59-10000 update failed keys from task: [u'test_collections-818', u'test_collections-979', u'test_collections-846', u'test_collections-512', u'test_collections-574', u'test_collections-541', u'test_collections-695'] 2021-05-03 02:01:36,104 | test | WARNING | Thread-8282 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:1YsVPgym-59-7000 update failed keys from task: [u'test_collections-728', u'test_collections-968', u'test_collections-705', u'test_collections-504', u'test_collections-866', u'test_collections-886', u'test_collections-641', u'test_collections-939', u'test_collections-537', u'test_collections-954', u'test_collections-533', u'test_collections-677', u'test_collections-895', u'test_collections-892', u'test_collections-794'] 2021-05-03 02:01:36,104 | test | WARNING | Thread-8293 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:eipH-59-6000 update failed keys from task: [u'test_collections-825', u'test_collections-634', u'test_collections-755', u'test_collections-710', u'test_collections-692', u'test_collections-671'] 2021-05-03 02:01:36,105 | test | WARNING | Thread-8295 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 update failed keys from task: [u'test_collections-579', u'test_collections-789', u'test_collections-663', u'test_collections-680'] 2021-05-03 02:01:36,107 | test | WARNING | Thread-8296 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:NI-59-7000 update failed keys from task: [u'test_collections-867'] 2021-05-03 02:01:36,132 | test | WARNING | Thread-8296 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:NI-59-7000 delete failed keys from task: [u'test_collections-178', u'test_collections-187'] 2021-05-03 02:01:36,142 | test | WARNING | Thread-8296 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:NI-59-7000 create failed keys from task: [u'test_collections-2503', u'test_collections-2517', u'test_collections-2605'] 2021-05-03 02:01:36,190 | test | WARNING | Thread-8295 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 delete failed keys from task: [u'test_collections-14', u'test_collections-61', u'test_collections-305', u'test_collections-111', u'test_collections-178', u'test_collections-82', u'test_collections-295'] 2021-05-03 02:01:36,209 | test | WARNING | Thread-8281 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:STVsE-59-10000 delete failed keys from task: [u'test_collections-59', u'test_collections-208', u'test_collections-74', u'test_collections-225', u'test_collections-324', u'test_collections-460', u'test_collections-48', u'test_collections-49', u'test_collections-398', u'test_collections-274', u'test_collections-471', u'test_collections-491'] 2021-05-03 02:01:36,223 | test | WARNING | Thread-8293 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:eipH-59-6000 delete failed keys from task: [u'test_collections-146', u'test_collections-322', u'test_collections-103', u'test_collections-378'] 2021-05-03 02:01:36,249 | test | WARNING | Thread-8294 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:TZYQVokZ7wPcamoYYwx-59-10000 update failed keys from task: [u'test_collections-867'] 2021-05-03 02:01:36,250 | test | WARNING | Thread-8299 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:N_PXW2o-59-8000 create failed keys from task: [u'test_collections-2502'] 2021-05-03 02:01:36,250 | test | WARNING | Thread-8292 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:cxthWt8E4gz2T8yu-59-9000 update failed keys from task: [u'test_collections-702', u'test_collections-821', u'test_collections-865', u'test_collections-630', u'test_collections-590'] 2021-05-03 02:01:36,257 | test | WARNING | Thread-8302 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F9s139WOKc2hKwtSJ4Hl5-59-11000 create failed keys from task: [u'test_collections-2634'] 2021-05-03 02:01:36,259 | test | WARNING | Thread-8295 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:ZyUYP%09RBdDJyy59s-9a-59-6000 create failed keys from task: [u'test_collections-2605', u'test_collections-2517'] 2021-05-03 02:01:36,260 | test | WARNING | Thread-8294 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:TZYQVokZ7wPcamoYYwx-59-10000 delete failed keys from task: [u'test_collections-178', u'test_collections-187'] 2021-05-03 02:01:36,263 | test | WARNING | Thread-8298 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:nJPk3GLaB751nVIcOqP4-59-6000 update failed keys from task: [u'test_collections-805', u'test_collections-843', u'test_collections-679', u'test_collections-789', u'test_collections-680'] 2021-05-03 02:01:36,270 | test | WARNING | Thread-8297 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:YzrzWdVpYCX_fqNzW-59-9000 update failed keys from task: [u'test_collections-549', u'test_collections-579', u'test_collections-843', u'test_collections-789', u'test_collections-680'] 2021-05-03 02:01:36,270 | test | WARNING | Thread-8301 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:W-59-9000 update failed keys from task: [u'test_collections-714', u'test_collections-711', u'test_collections-510', u'test_collections-741'] 2021-05-03 02:01:36,273 | test | WARNING | Thread-8305 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:6rbId54Fe-tLPHl4vN%Y6K-59-7000 delete failed keys from task: [u'test_collections-139'] 2021-05-03 02:01:36,292 | test | WARNING | Thread-8281 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:STVsE-59-10000 create failed keys from task: [u'test_collections-2706', u'test_collections-2742', u'test_collections-2658', u'test_collections-2553'] 2021-05-03 02:01:36,301 | test | WARNING | Thread-8307 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:D-VsvsD-59-10000 create failed keys from task: [u'test_collections-2541'] 2021-05-03 02:01:36,302 | test | WARNING | Thread-8292 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:cxthWt8E4gz2T8yu-59-9000 delete failed keys from task: [u'test_collections-36', u'test_collections-409', u'test_collections-73', u'test_collections-311', u'test_collections-354', u'test_collections-281'] 2021-05-03 02:01:36,305 | test | WARNING | Thread-8282 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:1YsVPgym-59-7000 delete failed keys from task: [u'test_collections-54', u'test_collections-408', u'test_collections-449', u'test_collections-128', u'test_collections-304', u'test_collections-96', u'test_collections-225', u'test_collections-445', u'test_collections-105', u'test_collections-204', u'test_collections-289', u'test_collections-267', u'test_collections-222', u'test_collections-260', u'test_collections-319', u'test_collections-456', u'test_collections-133', u'test_collections-233', u'test_collections-294', u'test_collections-394'] 2021-05-03 02:01:36,331 | test | WARNING | Thread-8294 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:TZYQVokZ7wPcamoYYwx-59-10000 create failed keys from task: [u'test_collections-2897'] 2021-05-03 02:01:36,345 | test | WARNING | Thread-8301 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:W-59-9000 delete failed keys from task: [u'test_collections-405', u'test_collections-424'] 2021-05-03 02:01:36,351 | test | WARNING | Thread-8292 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:cxthWt8E4gz2T8yu-59-9000 create failed keys from task: [u'test_collections-2512', u'test_collections-2863', u'test_collections-2627'] 2021-05-03 02:01:36,352 | test | WARNING | Thread-8298 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:nJPk3GLaB751nVIcOqP4-59-6000 delete failed keys from task: [u'test_collections-24'] 2021-05-03 02:01:36,358 | test | WARNING | Thread-8298 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:nJPk3GLaB751nVIcOqP4-59-6000 create failed keys from task: [u'test_collections-2890'] 2021-05-03 02:01:36,375 | test | WARNING | Thread-8297 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:YzrzWdVpYCX_fqNzW-59-9000 create failed keys from task: [u'test_collections-2589', u'test_collections-2890'] 2021-05-03 02:01:36,395 | test | WARNING | Thread-8282 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:1YsVPgym-59-7000 create failed keys from task: [u'test_collections-2548'] 2021-05-03 02:01:47,441 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 02:02:53,233 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 02:02:53,234 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 02:03:18,811 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 02:03:18,812 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-03 02:03:34,892 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 02:03:34,894 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 02:04:28,463 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 02:04:47,683 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.5510001183 seconds to complete 2021-05-03 02:04:47,700 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1878.669s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_3 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_4', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 4, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 02:04:59,096 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 02:05:00,096 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 02:05:00,973 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fde type=client open_count=1 channel=[id: 0x666c4013, 0.0.0.0/0.0.0.0:51661] timeout=300.0>) 2021-05-03 02:05:07,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fdf type=client open_count=1 channel=[id: 0x38cfe583, 0.0.0.0/0.0.0.0:55155] timeout=300.0>) 2021-05-03 02:05:09,584 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fe0 type=client open_count=1 channel=[id: 0x2fe5bf21, 0.0.0.0/0.0.0.0:34352] timeout=300.0>) 2021-05-03 02:05:11,605 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fe1 type=client open_count=1 channel=[id: 0xdeba7201, 0.0.0.0/0.0.0.0:50005] timeout=300.0>) 2021-05-03 02:05:13,628 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fe2 type=client open_count=1 channel=[id: 0xdc24a04d, 0.0.0.0/0.0.0.0:48408] timeout=300.0>) 2021-05-03 02:05:15,651 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fe3 type=client open_count=1 channel=[id: 0x8fca1d85, 0.0.0.0/0.0.0.0:58341] timeout=300.0>) 2021-05-03 02:05:17,676 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fe4 type=client open_count=1 channel=[id: 0xe038e87a, 0.0.0.0/0.0.0.0:45261] timeout=300.0>) 2021-05-03 02:05:19,710 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fe5 type=client open_count=1 channel=[id: 0x5fb54e7b, 0.0.0.0/0.0.0.0:34480] timeout=300.0>) 2021-05-03 02:05:21,732 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x117fe6 type=client open_count=1 channel=[id: 0xdfd25a93, 0.0.0.0/0.0.0.0:49999] timeout=300.0>) 2021-05-03 02:05:23,753 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fe7 type=client open_count=1 channel=[id: 0x5e0ca408, 0.0.0.0/0.0.0.0:55224] timeout=300.0>) 2021-05-03 02:05:31,561 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fe8 type=client open_count=1 channel=[id: 0x55bc4682, 0.0.0.0/0.0.0.0:42751] timeout=300.0>) 2021-05-03 02:05:33,582 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fe9 type=client open_count=1 channel=[id: 0x6f5109ed, 0.0.0.0/0.0.0.0:56526] timeout=300.0>) 2021-05-03 02:05:35,627 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fea type=client open_count=1 channel=[id: 0x67007264, 0.0.0.0/0.0.0.0:58909] timeout=300.0>) 2021-05-03 02:05:37,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117feb type=client open_count=1 channel=[id: 0xb88fee9f, 0.0.0.0/0.0.0.0:52324] timeout=300.0>) 2021-05-03 02:05:39,723 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fec type=client open_count=1 channel=[id: 0x4191ce73, 0.0.0.0/0.0.0.0:38253] timeout=300.0>) 2021-05-03 02:05:41,744 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fed type=client open_count=1 channel=[id: 0x00335436, 0.0.0.0/0.0.0.0:36155] timeout=300.0>) 2021-05-03 02:05:43,766 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fee type=client open_count=1 channel=[id: 0x4e7bc010, 0.0.0.0/0.0.0.0:59973] timeout=300.0>) 2021-05-03 02:05:45,789 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x117fef type=client open_count=1 channel=[id: 0xee851f82, 0.0.0.0/0.0.0.0:56409] timeout=300.0>) 2021-05-03 02:05:47,811 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff0 type=client open_count=1 channel=[id: 0x6884cdea, 0.0.0.0/0.0.0.0:56917] timeout=300.0>) 2021-05-03 02:05:55,453 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff1 type=client open_count=1 channel=[id: 0xa20e4fd9, 0.0.0.0/0.0.0.0:58258] timeout=300.0>) 2021-05-03 02:05:57,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff2 type=client open_count=1 channel=[id: 0xae82bcac, 0.0.0.0/0.0.0.0:60124] timeout=300.0>) 2021-05-03 02:05:59,502 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff3 type=client open_count=1 channel=[id: 0x5c4c40ad, 0.0.0.0/0.0.0.0:48995] timeout=300.0>) 2021-05-03 02:06:01,528 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff4 type=client open_count=1 channel=[id: 0x01002dc2, 0.0.0.0/0.0.0.0:49789] timeout=300.0>) 2021-05-03 02:06:03,552 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff5 type=client open_count=1 channel=[id: 0xb001b344, 0.0.0.0/0.0.0.0:37614] timeout=300.0>) 2021-05-03 02:06:05,594 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff6 type=client open_count=1 channel=[id: 0x6bd9eaf4, 0.0.0.0/0.0.0.0:40952] timeout=300.0>) 2021-05-03 02:06:07,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff7 type=client open_count=1 channel=[id: 0xed229d88, 0.0.0.0/0.0.0.0:53529] timeout=300.0>) 2021-05-03 02:06:09,651 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff8 type=client open_count=1 channel=[id: 0xbcaa78a4, 0.0.0.0/0.0.0.0:53775] timeout=300.0>) 2021-05-03 02:06:11,681 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ff9 type=client open_count=1 channel=[id: 0x3876b935, 0.0.0.0/0.0.0.0:39453] timeout=300.0>) 2021-05-03 02:06:13,703 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x117ffa type=client open_count=1 channel=[id: 0xe105d88b, 0.0.0.0/0.0.0.0:47429] timeout=300.0>) 2021-05-03 02:06:15,726 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x117ffb type=client open_count=1 channel=[id: 0x8c623259, 0.0.0.0/0.0.0.0:43164] timeout=300.0>) 2021-05-03 02:06:23,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x117ffc type=client open_count=1 channel=[id: 0xf12097a1, 0.0.0.0/0.0.0.0:45104] timeout=300.0>) 2021-05-03 02:06:25,796 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x117ffd type=client open_count=1 channel=[id: 0x90241b65, 0.0.0.0/0.0.0.0:57267] timeout=300.0>) 2021-05-03 02:06:27,823 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x117ffe type=client open_count=1 channel=[id: 0x3cc6d3ea, 0.0.0.0/0.0.0.0:42951] timeout=300.0>) 2021-05-03 02:06:29,845 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x117fff type=client open_count=1 channel=[id: 0x6500d13a, 0.0.0.0/0.0.0.0:33581] timeout=300.0>) 2021-05-03 02:06:31,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x118000 type=client open_count=1 channel=[id: 0xc2baaa0c, 0.0.0.0/0.0.0.0:46407] timeout=300.0>) 2021-05-03 02:06:33,885 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x118001 type=client open_count=1 channel=[id: 0xe3968c49, 0.0.0.0/0.0.0.0:50823] timeout=300.0>) 2021-05-03 02:06:35,910 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x118002 type=client open_count=1 channel=[id: 0x645917aa, 0.0.0.0/0.0.0.0:33881] timeout=300.0>) 2021-05-03 02:06:37,930 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x118003 type=client open_count=1 channel=[id: 0x45a1403f, 0.0.0.0/0.0.0.0:57345] timeout=300.0>) 2021-05-03 02:06:39,951 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x118004 type=client open_count=1 channel=[id: 0x0b68b098, 0.0.0.0/0.0.0.0:55111] timeout=300.0>) 2021-05-03 02:06:45,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x118005 type=client open_count=1 channel=[id: 0xa83bcf78, 0.0.0.0/0.0.0.0:41007] timeout=300.0>) 2021-05-03 02:06:47,286 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x118006 type=client open_count=1 channel=[id: 0xfa3c9eca, 0.0.0.0/0.0.0.0:42274] timeout=300.0>) 2021-05-03 02:06:49,302 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x118007 type=client open_count=1 channel=[id: 0x0dd92fdf, 0.0.0.0/0.0.0.0:34869] timeout=300.0>) 2021-05-03 02:06:51,323 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x118008 type=client open_count=1 channel=[id: 0xf25f0f5c, 0.0.0.0/0.0.0.0:41655] timeout=300.0>) 2021-05-03 02:06:53,371 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x118009 type=client open_count=1 channel=[id: 0x498207a2, 0.0.0.0/0.0.0.0:44133] timeout=300.0>) 2021-05-03 02:06:55,441 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:07:14,631 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 02:07:29,944 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 02:07:29,990 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 02:07:30,180 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 02:08:03,815 | test | INFO | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.32082598842 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:08:03,851 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:08:08,970 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:08:19,079 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:08:19,115 | test | INFO | pool-6-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.2999999523 sec 2021-05-03 02:08:19,711 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 02:08:59,696 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 02:08:59,986 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 631491792 | 69352220 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 562440176 | 102104941 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 649135008 | 40931217 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-03 02:08:59,986 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 02:10:05,700 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:10:05,717 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:25:51,361 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *........................................**.....................*............................X | 59.596 | | 43.801 | ................................................................*X | 43.801 | | 29.701 | ...........................................X | 29.701 | | 10.901 | ...............*X | 10.901 | | 0.000 | *************************.**............................................................................*.......................*X | 66.692 | | 65.092 | ................................................................................................X | 65.092 | | 49.906 | .........................................................................*X | 49.906 | | 38.206 | ........................................................*X | 38.206 | | 25.406 | .....................................X | 25.406 | | 4.600 | .....X | 4.600 | | 0.400 | *X | 0.400 | | 0.000 | **********************************************************************************.**.****...**.......*.............*X | 17.100 | | 15.600 | ......................*X | 15.600 | | 4.400 | .....*X | 4.400 | | 4.100 | .....*X | 4.100 | | 3.700 | ....X | 3.700 | | 1.300 | X | 1.300 | | 0.400 | *X | 0.400 | | 0.000 | **..**...*......X | 7.500 | | 6.000 | .......*X | 6.000 | | 4.400 | .....X | 4.400 | | 1.200 | *.*X | 1.500 | | 1.100 | X | 1.100 | | 0.300 | *X | 0.300 | | 0.000 | **************************************..*...**............*...................**...*X | 26.600 | | 18.600 | ..........................X | 18.600 | | 4.200 | .....*X | 4.200 | | 0.000 | *******************************************************************************************************************************************************************************..........**.................**.......................*...........................*X | 51.719 | | 40.000 | ..........................................................X | 40.000 | | 24.600 | ...................................X | 24.600 | | 0.000 | *****************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 02:25:53,046 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..........**....*......X | 31.497 | | 14.700 | ........*X | 14.700 | | 8.400 | ....X | 8.400 | | 0.000 | ******************************...........**..............................*...........................................................*X | 157.223 | | 155.423 | .................................................................................................*X | 155.423 | | 17.700 | ..........**X | 17.700 | | 1.500 | **.***X | 2.600 | | 2.600 | X | 2.600 | | 1.000 | X | 1.000 | | 0.400 | *X | 0.400 | | 0.000 | *******************************X | 1.500 | | 1.100 | X | 1.100 | | 0.300 | X | 0.300 | | 0.000 | *******.**.**X | 4.100 | | 1.700 | X | 1.700 | | 0.800 | *X | 0.800 | | 0.000 | ******************************.**...*.......**X | 17.700 | | 12.500 | ......X | 12.500 | | 2.100 | *X | 2.100 | | 0.000 | ************X | 1.500 | | 1.200 | *X | 1.200 | | 0.000 | ******************X | 1.500 | | 1.200 | X | 1.200 | | 0.800 | X | 0.800 | | 0.000 | **********X | 1.500 | | 1.200 | *X | 1.200 | | 0.000 | **********..*......**.........*................*X | 52.400 | | 45.500 | ...........................*X | 45.500 | | 2.800 | **X | 2.800 | | 0.400 | X | 0.400 | | 0.000 | ********************************************************************************************************************************************************************************************************X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:25:53,184 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***................*..........................*......................................................**....X | 1163.516 | | 1055.158 | .........................................................................................*X | 1055.158 | | 798.567 | ...................................................................X | 798.567 | | 764.190 | ................................................................*X | 764.190 | | 682.453 | .........................................................X | 682.453 | | 613.200 | ...................................................X | 613.200 | | 490.300 | .........................................**..**...................**.*X | 754.225 | | 748.125 | ...............................................................*.*.*X | 774.264 | | 766.064 | ................................................................X | 766.064 | | 741.700 | ..............................................................*X | 741.700 | | 726.400 | .............................................................*X | 726.400 | | 664.500 | ........................................................**X | 664.500 | | 619.500 | ....................................................X | 619.500 | | 466.042 | .......................................*X | 466.042 | | 423.542 | ...................................*X | 423.542 | | 413.954 | ..................................**..**.*.**..**..........**....**....**X | 697.025 | | 664.725 | ........................................................*X | 664.725 | | 600.612 | ..................................................**.**..X | 629.200 | | 591.100 | .................................................*X | 591.100 | | 567.700 | ...............................................*X | 567.700 | | 526.916 | ............................................X | 526.916 | | 518.898 | ...........................................*..**..*.....*X | 619.104 | | 600.904 | ..................................................*X | 600.904 | | 578.085 | ................................................X | 578.085 | | 476.800 | .......................................X | 476.800 | | 438.100 | ....................................*X | 438.100 | | 415.600 | ..................................X | 415.600 | | 401.800 | .................................**.**.........**..*X | 543.810 | | 526.310 | ............................................X | 526.310 | | 507.990 | ..........................................X | 507.990 | | 480.590 | ........................................*X | 480.590 | | 470.790 | .......................................X | 470.790 | | 469.500 | .......................................**..*....*.....**.*X | 612.000 | | 609.600 | ...................................................*.**X | 620.970 | | 608.114 | ...................................................X | 608.114 | | 562.871 | ...............................................*X | 562.871 | | 542.971 | .............................................*.*.....**.**.X | 633.800 | | 629.100 | .....................................................*X | 629.100 | | 625.200 | ....................................................X | 625.200 | | 570.300 | ................................................X | 570.300 | | 451.300 | .....................................*X | 451.300 | | 401.400 | .................................***..*.**..***..*X | 477.946 | | 469.346 | .......................................*.**.*.*....***X | 556.500 | | 552.200 | ..............................................***X | 557.400 | | 531.300 | ............................................X | 531.300 | | 506.662 | ..........................................X | 506.662 | | 496.562 | .........................................*X | 496.562 | | 487.562 | ........................................X | 487.562 | | 449.530 | .....................................X | 449.530 | | 404.530 | .................................*X | 404.530 | | 400.030 | .................................X | 400.030 | | 390.100 | ................................*..*X | 415.200 | | 411.000 | ..................................X | 411.000 | | 409.800 | ..................................***..*.......**...**......***X | 623.819 | | 547.800 | ..............................................*X | 547.800 | | 536.700 | .............................................*X | 536.700 | | 529.500 | ............................................*..*X | 550.900 | | 511.180 | ..........................................X | 511.180 | | 471.080 | .......................................X | 471.080 | | 397.943 | .................................**.**..*.***.*.....**.**.X | 542.559 | | 536.959 | .............................................***.*.***.*.*X | 586.942 | | 564.500 | ...............................................X | 564.500 | | 514.100 | ...........................................X | 514.100 | | 387.300 | ................................*X | 387.300 | | 369.700 | ..............................**....****.*....****........***...X | 602.431 | | 552.431 | ..............................................X | 552.431 | | 381.388 | ...............................*X | 381.388 | | 254.088 | ....................*X | 254.088 | | 27.400 | .**X | 27.400 | | 0.000 | *..*...............**.............**....................*X | 588.282 | | 545.682 | .............................................*X | 545.682 | | 497.952 | .........................................X | 497.952 | | 471.858 | .......................................**.*X | 484.558 | | 479.513 | ........................................X | 479.513 | | 477.260 | ........................................X | 477.260 | | 472.760 | .......................................**..*..*.....**.*...*X | 623.440 | | 599.454 | ..................................................**X | 603.354 | | 595.343 | ..................................................*.**..*.*X | 641.000 | | 613.460 | ...................................................*X | 613.460 | | 543.260 | .............................................X | 543.260 | | 466.425 | .......................................X | 466.425 | | 444.498 | .....................................**.*X | 459.298 | | 448.566 | .....................................*...**..*...X | 545.883 | | 507.883 | ..........................................*X | 507.883 | | 473.683 | .......................................*X | 473.683 | | 445.008 | .....................................*...**...*.....*.*X | 584.200 | | 582.400 | .................................................*X | 582.400 | | 564.500 | ...............................................X | 564.500 | | 557.200 | ..............................................**....**X | 604.553 | | 604.384 | ..................................................X | 604.384 | | 577.485 | ................................................X | 577.485 | | 487.935 | ........................................*X | 487.935 | | 469.763 | .......................................X | 469.763 | | 459.153 | ......................................*X | 459.153 | | 444.950 | .....................................***.*.....X | 523.000 | | 511.600 | ..........................................*X | 511.600 | | 497.900 | .........................................X | 497.900 | | 457.900 | ......................................*X | 457.900 | | 457.400 | ......................................**........**...*X | 582.000 | | 570.400 | ................................................*.*X | 590.700 | | 585.000 | .................................................*X | 585.000 | | 567.571 | ...............................................***X | 568.871 | | 533.971 | ............................................X | 533.971 | | 471.621 | .......................................*X | 471.621 | | 431.439 | ....................................**.***.*..X | 480.595 | | 476.959 | .......................................**..**..**.*X | 525.000 | | 522.500 | ...........................................**...***....*..***X | 628.210 | | 580.286 | ................................................**.*X | 588.986 | | 562.918 | ...............................................X | 562.918 | | 543.832 | .............................................*X | 543.832 | | 466.632 | .......................................*X | 466.632 | | 447.241 | .....................................X | 447.241 | | 369.941 | ..............................*..**.*....****..***..*X | 494.784 | | 487.249 | ........................................*...***....**X | 565.223 | | 560.162 | ...............................................*X | 560.162 | | 540.116 | .............................................*.*.**.*..***X | 601.600 | | 511.379 | ..........................................**X | 511.379 | | 482.197 | ........................................X | 482.197 | | 453.597 | .....................................X | 453.597 | | 448.642 | .....................................*X | 448.642 | | 447.705 | .....................................*..*....**...**.....X | 605.652 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 02:25:53,744 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 45.909805978 | 3693387776 | 2534666240 | 0 / 3909087232 | 126675 / 254115 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 47.3364978903 | 3693387776 | 2635735040 | 0 / 3909087232 | 125085 / 253992 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 47.459375806 | 6078930944 | 4821143552 | 0 / 3758092288 | 126675 / 247786 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 51.2377514183 | 4201836544 | 3093733376 | 0 / 3758092288 | 124604 / 250506 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 02:26:08,204 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 02:27:59,279 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155353024 | 240800763 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 156068992 | 266589861 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 435968432 | 460326620 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-03 02:27:59,280 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 02:28:10,721 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_in_out 2021-05-03 02:28:10,723 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_in_out 2021-05-03 02:28:18,500 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:28:18,880 | test | INFO | pool-6-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 23.2575950983 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 22.109016811 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 15.0227617602 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 0.803212851406 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 16.8487928844 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-03 02:28:18,944 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:28:24,140 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:28:30,160 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:28:32,891 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 1.09 2021-05-03 02:28:38,122 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 9.01 2021-05-03 02:28:43,286 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 13.51 2021-05-03 02:28:48,424 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 18.11 2021-05-03 02:28:53,551 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 24.49 2021-05-03 02:28:58,671 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-03 02:29:03,783 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 02:29:08,869 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 02:29:14,604 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 34.3 2021-05-03 02:29:18,421 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in bucket2:Lf28a7OuWx4-24-710000:q8g7mtNZ2QjrG-29-885000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1142 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:18,421 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in bucket2:Lf28a7OuWx4-24-710000:q8g7mtNZ2QjrG-29-885000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1142 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:19,950 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 40.18 2021-05-03 02:29:25,118 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 44.82 2021-05-03 02:29:25,609 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in bucket2:Lf28a7OuWx4-24-710000:q8g7mtNZ2QjrG-29-885000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:25,611 | test | INFO | pool-6-thread-15 | [table_view:display:72] Keys failed in bucket2:Lf28a7OuWx4-24-710000:q8g7mtNZ2QjrG-29-885000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:30,328 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 50.14 2021-05-03 02:29:35,571 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 55.76 2021-05-03 02:29:40,724 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 61.04 2021-05-03 02:29:43,266 | test | INFO | pool-6-thread-8 | [table_view:display:72] Keys failed in bucket2:-5sfKCfwSvDcHjlp0p-26-693000:RC-26-829000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-910 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-880 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:43,269 | test | INFO | pool-6-thread-8 | [table_view:display:72] Keys failed in bucket2:-5sfKCfwSvDcHjlp0p-26-693000:RC-26-829000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-910 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-880 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:46,015 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 65.62 2021-05-03 02:29:51,216 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 02:29:54,104 | test | INFO | pool-6-thread-2 | [table_view:display:72] Keys failed in bucket2:BKJSx6H-25-897000:tR9c5H-Ht53ydhoO0z-26-225000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1389 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1219 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:54,105 | test | INFO | pool-6-thread-2 | [table_view:display:72] Keys failed in bucket2:BKJSx6H-25-897000:tR9c5H-Ht53ydhoO0z-26-225000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1389 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1219 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:29:56,349 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 02:30:01,490 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 68.76 2021-05-03 02:30:06,711 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 73.59 2021-05-03 02:30:11,885 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 78.54 2021-05-03 02:30:17,069 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 83.72 2021-05-03 02:30:22,190 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 89.32 2021-05-03 02:30:27,440 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 94.79 2021-05-03 02:30:27,773 | test | INFO | pool-6-thread-30 | [table_view:display:72] Keys failed in default:yn%ii-19-823000:OU6-29-884000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1136 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:30:27,775 | test | INFO | pool-6-thread-30 | [table_view:display:72] Keys failed in default:yn%ii-19-823000:OU6-29-884000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1136 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:30:32,589 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 99.01 2021-05-03 02:30:37,786 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:30:42,918 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:30:45,609 | test | INFO | pool-6-thread-6 | [table_view:display:72] Keys failed in default:yn%ii-19-823000:h4%eAPP6HChPXBQ4-19-824000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-608 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:30:53,071 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173eea type=client open_count=1 channel=[id: 0x6cc743da, 0.0.0.0/0.0.0.0:39792] timeout=300.0>) 2021-05-03 02:30:53,135 | test | ERROR | pool-6-thread-18 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:30:55,596 | test | INFO | pool-6-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 156.717999935 sec 2021-05-03 02:32:19,513 | test | ERROR | pool-6-thread-19 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 02:32:19,515 | test | ERROR | pool-6-thread-19 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 02:32:19,516 | test | ERROR | pool-6-thread-19 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 02:34:10,296 | test | ERROR | pool-6-thread-19 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 02:34:10,302 | test | ERROR | pool-6-thread-19 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 02:37:13,306 | test | INFO | pool-6-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..*..**.*X | 25.602 | | 14.402 | .X | 14.402 | | 8.194 | *......**...............**...........................*..............................................*.....X | 482.855 | | 409.900 | ...................................................................................X | 409.900 | | 285.657 | ..........................................................*X | 285.657 | | 253.657 | ...................................................X | 253.657 | | 239.157 | ................................................X | 239.157 | | 175.724 | ...................................X | 175.724 | | 149.324 | .............................*X | 149.324 | | 133.589 | ..........................*X | 133.589 | | 123.189 | ........................X | 123.189 | | 115.489 | ......................X | 115.489 | | 95.703 | ..................**..**...............***...X | 191.076 | | 163.176 | ................................X | 163.176 | | 157.921 | ...............................*X | 157.921 | | 153.121 | ..............................X | 153.121 | | 151.521 | ..............................X | 151.521 | | 151.500 | ..............................X | 151.500 | | 132.300 | ..........................X | 132.300 | | 119.400 | .......................*X | 119.400 | | 114.400 | ......................**......***..X | 153.400 | | 148.948 | .............................X | 148.948 | | 145.148 | .............................*X | 145.148 | | 130.148 | .........................*X | 130.148 | | 115.230 | ......................X | 115.230 | | 103.002 | ....................*..**..*.*X | 128.634 | | 123.634 | ........................**...*.**..X | 152.700 | | 132.700 | ..........................*.....*X | 156.400 | | 154.600 | ...............................**.....**..*X | 188.388 | | 173.409 | ..................................**.....*..**.....*.....**....*X | 271.474 | | 256.474 | ....................................................X | 256.474 | | 219.916 | ............................................X | 219.916 | | 169.901 | ..................................*X | 169.901 | | 134.901 | ..........................*X | 134.901 | | 73.785 | ..............X | 73.785 | | 53.803 | ..........*X | 53.803 | | 38.803 | .......X | 38.803 | | 5.003 | *X | 5.003 | | 0.000 | ***************.....**.*X | 31.496 | | 20.996 | ...X | 20.996 | | 6.996 | *X | 6.996 | | 0.000 | ***************************************************.*...**..*X | 32.000 | | 22.600 | ...*X | 22.600 | | 13.400 | .*X | 13.400 | | 5.000 | X | 5.000 | | 0.400 | *X | 0.400 | | 0.000 | ********************X | 0.500 | | 0.300 | *X | 0.300 | | 0.100 | X | 0.100 | | 0.000 | **************X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:37:15,073 | test | INFO | pool-6-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***********..**........*X | 35.005 | | 29.005 | .......**...............**..................*............*....................X | 243.840 | | 241.240 | ........................................................................*X | 241.240 | | 209.840 | ..............................................................X | 209.840 | | 189.500 | ........................................................X | 189.500 | | 168.100 | .................................................**.....X | 183.100 | | 162.600 | ................................................**...X | 172.600 | | 167.600 | .................................................*.....*.**....*..........*...*X | 242.036 | | 215.536 | ................................................................X | 215.536 | | 178.736 | .....................................................X | 178.736 | | 148.700 | ............................................*......X | 171.100 | | 153.500 | .............................................**...........*...*..................................X | 311.500 | | 306.500 | ...........................................................................................**........*X | 329.865 | | 269.865 | ................................................................................*.........*X | 299.865 | | 270.090 | ................................................................................*.........*X | 299.401 | | 269.401 | ................................................................................*X | 269.401 | | 232.853 | .....................................................................*****...............**........**X | 309.400 | | 230.800 | ....................................................................X | 230.800 | | 148.400 | ...........................................*X | 148.400 | | 88.400 | .........................X | 88.400 | | 3.800 | *X | 3.800 | | 0.000 | *************************************************************************************************..*....**...*X | 31.400 | | 26.000 | ......*X | 26.000 | | 15.600 | ...*X | 15.600 | | 7.700 | .X | 7.700 | | 0.000 | ****************************************X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:37:15,299 | test | INFO | pool-6-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +----------+-------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ********************......*.............*.........**X | 366.455 | | 354.651 | ..........................X | 354.651 | | 306.151 | .......................*X | 306.151 | | 187.829 | .............**X | 187.829 | | 158.629 | ...........X | 158.629 | | 143.900 | ..........*X | 143.900 | | 140.179 | .........**...**.*X | 186.429 | | 159.729 | ...........X | 159.729 | | 131.229 | .........**...*.....*X | 232.500 | | 212.900 | ...............X | 212.900 | | 195.900 | ..............X | 195.900 | | 128.200 | .........*X | 128.200 | | 127.700 | .........X | 127.700 | | 122.146 | ........*.**...**...*......X | 290.331 | | 257.931 | ...................*X | 257.931 | | 251.723 | ..................X | 251.723 | | 208.442 | ...............**X | 214.142 | | 181.342 | .............*X | 181.342 | | 156.000 | ...........X | 156.000 | | 155.300 | ...........**....***...*...*....*X | 335.486 | | 313.086 | .......................**..X | 335.619 | | 333.619 | .........................**X | 341.019 | | 332.500 | .........................*X | 332.500 | | 329.766 | ........................X | 329.766 | | 329.366 | ........................**...*..**...**..............***....*..X | 688.708 | | 687.474 | ....................................................*X | 687.474 | | 677.167 | ....................................................**...............**.......*X | 961.108 | | 942.208 | ........................................................................X | 942.208 | | 860.610 | ..................................................................***....*..................**X | 1137.900 | | 1095.800 | ....................................................................................X | 1095.800 | | 968.000 | ..........................................................................X | 968.000 | | 719.100 | .......................................................*X | 719.100 | | 654.407 | ..................................................**.....................***.........X | 1036.620 | | 951.880 | .........................................................................X | 951.880 | | 803.480 | .............................................................*X | 803.480 | | 688.118 | ....................................................**.......X | 775.776 | | 775.776 | ...........................................................*......*X | 851.276 | | 774.700 | ...........................................................X | 774.700 | | 722.112 | .......................................................*X | 722.112 | | 670.312 | ...................................................**............................***.........X | 1144.757 | | 1113.058 | ......................................................................................*.X | 1123.090 | | 1115.790 | ......................................................................................**..X | 1144.837 | | 1091.741 | ....................................................................................*X | 1091.741 | | 1034.871 | ................................................................................*X | 1034.871 | | 916.840 | ......................................................................**.*X | 929.184 | | 914.073 | ......................................................................X | 914.073 | | 855.373 | ..................................................................*X | 855.373 | | 795.263 | .............................................................**...*......*.....*X | 976.802 | | 916.402 | ......................................................................X | 916.402 | | 835.789 | ................................................................*X | 835.789 | | 834.140 | ................................................................**.......*...........**..X | 1091.400 | | 1061.700 | ..................................................................................X | 1061.700 | | 1043.600 | ................................................................................*X | 1043.600 | | 952.500 | .........................................................................**...*....**............*X | 1190.800 | | 936.801 | ........................................................................*X | 936.801 | | 806.401 | ..............................................................*X | 806.401 | | 799.695 | .............................................................X | 799.695 | | 764.844 | ..........................................................*X | 764.844 | | 704.644 | ......................................................*X | 704.644 | | 700.849 | .....................................................*.*..........X | 832.001 | | 832.001 | ................................................................X | 832.001 | | 822.501 | ...............................................................X | 822.501 | | 804.936 | ..............................................................*X | 804.936 | | 699.936 | .....................................................**..*.**..*.*.....**........**...........*................*X | 1275.460 | | 1273.891 | ..................................................................................................X | 1273.891 | | 1177.733 | ...........................................................................................X | 1177.733 | | 1166.733 | ..........................................................................................*X | 1166.733 | | 1141.927 | ........................................................................................**..X | 1163.026 | | 952.526 | .........................................................................X | 952.526 | | 948.526 | .........................................................................*X | 948.526 | | 795.826 | .............................................................X | 795.826 | | 627.826 | ................................................X | 627.826 | +----------+-------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 02:37:15,880 | test | WARNING | Thread-11107 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Lf28a7OuWx4-24-710000:q8g7mtNZ2QjrG-29-885000 create failed keys from task: [u'test_collections-1142', u'test_collections-1231'] 2021-05-03 02:37:15,880 | test | WARNING | Thread-11109 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:BKJSx6H-25-897000:tR9c5H-Ht53ydhoO0z-26-225000 create failed keys from task: [u'test_collections-1389', u'test_collections-1219'] 2021-05-03 02:37:15,937 | test | WARNING | Thread-11069 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:-5sfKCfwSvDcHjlp0p-26-693000:RC-26-829000 create failed keys from task: [u'test_collections-910', u'test_collections-880'] 2021-05-03 02:37:16,250 | test | WARNING | Thread-11488 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:yn%ii-19-823000:OU6-29-884000 create failed keys from task: [u'test_collections-1136'] 2021-05-03 02:37:16,256 | test | WARNING | Thread-11495 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:yn%ii-19-823000:h4%eAPP6HChPXBQ4-19-824000 update failed keys from task: [u'test_collections-608'] 2021-05-03 02:37:28,944 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 02:38:52,125 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 02:38:52,127 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 02:39:17,726 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 02:39:17,726 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-03 02:39:40,506 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 02:39:40,506 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 02:40:28,694 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 02:40:47,127 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1460001469 seconds to complete 2021-05-03 02:40:47,145 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2159.229s During the test, Remote Connections: 162, Disconnections: 162 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_4 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_5', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 5, 'bulk_api_crud': 'True'} test_rebalance_cycles (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 02:40:57,229 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #5 test_rebalance_cycles ========= 2021-05-03 02:40:58,450 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #5 test_rebalance_cycles ========= 2021-05-03 02:40:58,957 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748e3 type=client open_count=1 channel=[id: 0xd620d233, 0.0.0.0/0.0.0.0:46989] timeout=300.0>) 2021-05-03 02:41:04,654 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748e4 type=client open_count=1 channel=[id: 0xffc3ae75, 0.0.0.0/0.0.0.0:39886] timeout=300.0>) 2021-05-03 02:41:06,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748e5 type=client open_count=1 channel=[id: 0x64e59d61, 0.0.0.0/0.0.0.0:48161] timeout=300.0>) 2021-05-03 02:41:08,801 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748e6 type=client open_count=1 channel=[id: 0x29e0a16d, 0.0.0.0/0.0.0.0:56641] timeout=300.0>) 2021-05-03 02:41:10,822 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748e7 type=client open_count=1 channel=[id: 0xc9b00a7a, 0.0.0.0/0.0.0.0:33659] timeout=300.0>) 2021-05-03 02:41:12,846 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748e8 type=client open_count=1 channel=[id: 0xedfd9fed, 0.0.0.0/0.0.0.0:42325] timeout=300.0>) 2021-05-03 02:41:14,875 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748e9 type=client open_count=1 channel=[id: 0xe6f8c0fe, 0.0.0.0/0.0.0.0:60186] timeout=300.0>) 2021-05-03 02:41:16,897 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748ea type=client open_count=1 channel=[id: 0xcb81e614, 0.0.0.0/0.0.0.0:38905] timeout=300.0>) 2021-05-03 02:41:18,940 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1748eb type=client open_count=1 channel=[id: 0xd59c1f28, 0.0.0.0/0.0.0.0:43309] timeout=300.0>) 2021-05-03 02:41:20,967 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748ec type=client open_count=1 channel=[id: 0xe5fc904a, 0.0.0.0/0.0.0.0:56863] timeout=300.0>) 2021-05-03 02:41:28,684 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748ed type=client open_count=1 channel=[id: 0x072b90c8, 0.0.0.0/0.0.0.0:56792] timeout=300.0>) 2021-05-03 02:41:30,707 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748ee type=client open_count=1 channel=[id: 0x7af18184, 0.0.0.0/0.0.0.0:43175] timeout=300.0>) 2021-05-03 02:41:32,733 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748ef type=client open_count=1 channel=[id: 0xd7d059fb, 0.0.0.0/0.0.0.0:38989] timeout=300.0>) 2021-05-03 02:41:34,755 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748f0 type=client open_count=1 channel=[id: 0xc3b62f9c, 0.0.0.0/0.0.0.0:33169] timeout=300.0>) 2021-05-03 02:41:36,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748f1 type=client open_count=1 channel=[id: 0x535c4549, 0.0.0.0/0.0.0.0:44197] timeout=300.0>) 2021-05-03 02:41:38,813 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748f2 type=client open_count=1 channel=[id: 0x96eb91c9, 0.0.0.0/0.0.0.0:37971] timeout=300.0>) 2021-05-03 02:41:40,862 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748f3 type=client open_count=1 channel=[id: 0xaba44f71, 0.0.0.0/0.0.0.0:35387] timeout=300.0>) 2021-05-03 02:41:42,885 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748f4 type=client open_count=1 channel=[id: 0x32006485, 0.0.0.0/0.0.0.0:33495] timeout=300.0>) 2021-05-03 02:41:44,910 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1748f5 type=client open_count=1 channel=[id: 0xf190131e, 0.0.0.0/0.0.0.0:52487] timeout=300.0>) 2021-05-03 02:41:46,933 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748f6 type=client open_count=1 channel=[id: 0x1587d067, 0.0.0.0/0.0.0.0:35147] timeout=300.0>) 2021-05-03 02:41:54,727 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748f7 type=client open_count=1 channel=[id: 0xe6aa667b, 0.0.0.0/0.0.0.0:49697] timeout=300.0>) 2021-05-03 02:41:56,753 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748f8 type=client open_count=1 channel=[id: 0xd04e2b2b, 0.0.0.0/0.0.0.0:53275] timeout=300.0>) 2021-05-03 02:41:58,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748f9 type=client open_count=1 channel=[id: 0xc2ebf524, 0.0.0.0/0.0.0.0:52111] timeout=300.0>) 2021-05-03 02:42:00,796 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748fa type=client open_count=1 channel=[id: 0x9fa7a329, 0.0.0.0/0.0.0.0:40123] timeout=300.0>) 2021-05-03 02:42:02,819 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748fb type=client open_count=1 channel=[id: 0x83f7f9cb, 0.0.0.0/0.0.0.0:43434] timeout=300.0>) 2021-05-03 02:42:04,842 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748fc type=client open_count=1 channel=[id: 0xa4556779, 0.0.0.0/0.0.0.0:51503] timeout=300.0>) 2021-05-03 02:42:06,875 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748fd type=client open_count=1 channel=[id: 0x7f9d3a01, 0.0.0.0/0.0.0.0:35321] timeout=300.0>) 2021-05-03 02:42:08,910 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748fe type=client open_count=1 channel=[id: 0xadc59bc7, 0.0.0.0/0.0.0.0:44451] timeout=300.0>) 2021-05-03 02:42:10,930 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1748ff type=client open_count=1 channel=[id: 0xb2608c39, 0.0.0.0/0.0.0.0:58719] timeout=300.0>) 2021-05-03 02:42:12,954 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174900 type=client open_count=1 channel=[id: 0x41cc0455, 0.0.0.0/0.0.0.0:33163] timeout=300.0>) 2021-05-03 02:42:21,921 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174901 type=client open_count=1 channel=[id: 0xeacdaed5, 0.0.0.0/0.0.0.0:43597] timeout=300.0>) 2021-05-03 02:42:23,944 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174902 type=client open_count=1 channel=[id: 0x06d6690d, 0.0.0.0/0.0.0.0:50921] timeout=300.0>) 2021-05-03 02:42:25,967 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174903 type=client open_count=1 channel=[id: 0x370ad102, 0.0.0.0/0.0.0.0:57523] timeout=300.0>) 2021-05-03 02:42:27,990 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174904 type=client open_count=1 channel=[id: 0xb8d596fb, 0.0.0.0/0.0.0.0:42531] timeout=300.0>) 2021-05-03 02:42:30,016 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174905 type=client open_count=1 channel=[id: 0x4f650a27, 0.0.0.0/0.0.0.0:52353] timeout=300.0>) 2021-05-03 02:42:32,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174906 type=client open_count=1 channel=[id: 0xaaf41c57, 0.0.0.0/0.0.0.0:55984] timeout=300.0>) 2021-05-03 02:42:34,091 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174907 type=client open_count=1 channel=[id: 0x6a0e62a7, 0.0.0.0/0.0.0.0:53665] timeout=300.0>) 2021-05-03 02:42:36,117 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174908 type=client open_count=1 channel=[id: 0xea86c20a, 0.0.0.0/0.0.0.0:41259] timeout=300.0>) 2021-05-03 02:42:38,132 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x174909 type=client open_count=1 channel=[id: 0x64e0e9d8, 0.0.0.0/0.0.0.0:40783] timeout=300.0>) 2021-05-03 02:42:43,918 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17490a type=client open_count=1 channel=[id: 0xcaf3a3a9, 0.0.0.0/0.0.0.0:49899] timeout=300.0>) 2021-05-03 02:42:45,954 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17490b type=client open_count=1 channel=[id: 0xa44b1e1e, 0.0.0.0/0.0.0.0:53277] timeout=300.0>) 2021-05-03 02:42:47,984 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17490c type=client open_count=1 channel=[id: 0x6d5f3951, 0.0.0.0/0.0.0.0:57387] timeout=300.0>) 2021-05-03 02:42:50,022 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17490d type=client open_count=1 channel=[id: 0x6b32876a, 0.0.0.0/0.0.0.0:52291] timeout=300.0>) 2021-05-03 02:42:52,197 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17490e type=client open_count=1 channel=[id: 0x22b6c3f3, 0.0.0.0/0.0.0.0:38749] timeout=300.0>) 2021-05-03 02:42:54,219 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:43:11,872 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 02:43:26,900 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 02:43:26,954 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #5 test_rebalance_cycles ========= 2021-05-03 02:43:27,007 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #5 test_rebalance_cycles ========= 2021-05-03 02:43:58,744 | test | INFO | pool-7-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 8.3459404942 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-03 02:43:58,769 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:44:03,829 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:44:13,931 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:44:13,982 | test | INFO | pool-7-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.2380001545 sec 2021-05-03 02:44:14,476 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 02:44:14,762 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 02:44:14,947 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | olZHb-14-553000 | couchbase | 2 | none | 0 | 0 | 10825498624 | 0 | 0 | +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-03 02:44:14,947 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 02:44:16,716 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:44:16,726 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:44:44,385 | test | INFO | pool-7-thread-9 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 100.000 | ......................*..................................................**..........................**.X | 433.022 | | 423.422 | ................................................................................................X | 423.422 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:44:44,450 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 21.5846275336 | 3693387776 | 2985422848 | 0 / 3909087232 | 2090 / 4229 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 22.2541841004 | 3693387776 | 3062407168 | 0 / 3909087232 | 1599 / 3198 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 21.6500128107 | 6078930944 | 5241253888 | 0 / 3758092288 | 2201 / 4361 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 26.9520103761 | 4201836544 | 3574251520 | 0 / 3758092288 | 2230 / 4474 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 02:44:50,627 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 02:45:15,088 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | olZHb-14-553000 | couchbase | 2 | none | 0 | 10000 | 10825498624 | 103344576 | 177808814 | +-----------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 02:45:15,088 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #5 test_rebalance_cycles ========= 2021-05-03 02:45:23,230 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 0 2021-05-03 02:45:23,313 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:45:23,322 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:45:23,361 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:45:25,434 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:45:27,499 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:45:29,536 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:45:31,466 | test | INFO | pool-7-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 16.594991146 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 9.78809283552 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 9.01536136993 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 10.2025316456 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:45:31,579 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:45:31,808 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:45:36,713 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:45:38,642 | test | INFO | pool-7-thread-19 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.............*..........................*.......................................................**......X | 531.600 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:45:41,855 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 10.45 2021-05-03 02:45:46,980 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 34.12 2021-05-03 02:45:52,088 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.85 2021-05-03 02:45:57,256 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 93.72 2021-05-03 02:46:07,411 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:46:07,447 | test | INFO | pool-7-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 35.9830000401 sec 2021-05-03 02:46:07,499 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:46:07,506 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:46:07,549 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:46:07,644 | test | INFO | pool-7-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 83.2211289487 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 29.3554232462 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 36.9817151687 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 13.7204522097 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 31.8698347107 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:46:07,677 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:46:10,599 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2523 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:10,654 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4420 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4352 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:10,884 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:46:11,513 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:11,515 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12012 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:12,071 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2711 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2681 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:12,428 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4649 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:12,779 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 7.2 2021-05-03 02:46:15,282 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:15,285 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:16,173 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4778 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4875 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:16,650 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2907 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2897 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2984 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-2969 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-2814 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-2836 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:18,017 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:18,019 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12576 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:18,063 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 24.15 2021-05-03 02:46:19,194 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3119 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3089 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:19,246 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4959 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:20,418 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3330 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:21,477 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5190 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5260 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5217 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5234 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:22,628 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3491 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-3501 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-3542 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:23,201 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 36.27 2021-05-03 02:46:26,970 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5458 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:27,615 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3712 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3682 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3761 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3775 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:28,309 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 46.02 2021-05-03 02:46:30,697 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5720 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:46:30,700 | test | ERROR | pool-7-thread-25 | [task:call:3825] Failed to load 13 docs from 3920 to 5763 2021-05-03 02:46:30,700 | test | ERROR | pool-7-thread-25 | [task:call:3825] Failed to load 20 docs from 2000 to 3920 2021-05-03 02:46:30,700 | test | ERROR | pool-7-thread-25 | [task:call:3825] Failed to load 5 docs from 11600 to 13136 2021-05-03 02:46:31,888 | test | INFO | pool-7-thread-26 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..........................................................................**..........................*X | 372.872 | | 265.918 | ......................................................................*X | 265.918 | | 230.618 | ............................................................*X | 230.618 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:46:33,451 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 58.06 2021-05-03 02:46:38,556 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 72.09 2021-05-03 02:46:43,703 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 84.62 2021-05-03 02:46:48,861 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 99.75 2021-05-03 02:46:53,976 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:46:59,088 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:47:09,285 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x174f85 type=client open_count=1 channel=[id: 0x3d712ace, 0.0.0.0/0.0.0.0:59391] timeout=300.0>) 2021-05-03 02:47:09,331 | test | ERROR | pool-7-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:47:11,489 | test | INFO | pool-7-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 63.8450000286 sec 2021-05-03 02:47:11,490 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 1 2021-05-03 02:47:11,559 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:47:11,568 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:47:11,625 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:47:13,663 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:47:15,698 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:47:17,732 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:47:19,740 | test | INFO | pool-7-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 8.97110998479 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 6.85902303214 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 8.73860182371 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:47:19,780 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:47:19,888 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:47:24,864 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:47:29,987 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 5.16 2021-05-03 02:47:31,252 | test | INFO | pool-7-thread-2 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7085 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7122 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7115 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-6981 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-6998 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:47:34,594 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5493 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5480 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-5439 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5503 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5510 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-5378 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:47:35,117 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 18.01 2021-05-03 02:47:38,230 | test | INFO | pool-7-thread-7 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14399 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-14446 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:47:38,232 | test | INFO | pool-7-thread-7 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14399 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-14446 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:47:38,332 | test | INFO | pool-7-thread-2 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7260 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7254 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7199 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7165 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7251 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7236 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7211 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7229 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:47:39,819 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5704 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5605 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5657 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5558 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5618 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5596 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5640 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5520 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5694 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5612 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:47:40,240 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 34.96 2021-05-03 02:47:40,851 | test | ERROR | pool-7-thread-1 | [task:call:3825] Failed to load 13 docs from 5763 to 7532 2021-05-03 02:47:41,687 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5741 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:47:41,687 | test | ERROR | pool-7-thread-1 | [task:call:3825] Failed to load 17 docs from 3920 to 5763 2021-05-03 02:47:41,688 | test | ERROR | pool-7-thread-1 | [task:call:3825] Failed to load 2 docs from 13136 to 14610 2021-05-03 02:47:42,946 | test | INFO | pool-7-thread-3 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..........................*....................................*......................................*X | 379.900 | | 347.100 | ..........................................................................................*X | 347.100 | | 262.137 | ....................................................................X | 262.137 | | 167.223 | ...........................................*X | 167.223 | | 147.523 | .....................................X | 147.523 | | 128.325 | ................................X | 128.325 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:47:45,335 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 55.15 2021-05-03 02:47:50,447 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 76.17 2021-05-03 02:47:55,663 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 99.26 2021-05-03 02:48:05,852 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:48:05,894 | test | INFO | pool-7-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 46.1559998989 sec 2021-05-03 02:48:05,976 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:48:05,994 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:48:06,085 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:48:06,210 | test | INFO | pool-7-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 80.0883805563 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 17.0038412292 | Cluster node | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 27.0526315789 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 19.6173469388 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:48:06,308 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:48:09,016 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7753 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:09,035 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6072 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:09,486 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:48:11,457 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6291 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6251 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6301 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:11,532 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 8.59 2021-05-03 02:48:14,937 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15141 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15136 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:14,937 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15141 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15136 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:16,690 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 23.86 2021-05-03 02:48:17,043 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8162 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8177 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:17,565 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6494 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6504 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6425 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6557 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6423 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6533 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:21,519 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8348 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8457 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:21,838 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 35.27 2021-05-03 02:48:25,937 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15561 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15596 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15527 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:25,937 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15561 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15596 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15527 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:26,986 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 48.66 2021-05-03 02:48:28,447 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6795 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6786 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:28,776 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8683 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8640 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8536 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8713 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8696 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:29,832 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15733 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:29,835 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15733 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:31,546 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8737 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8799 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8745 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8764 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:32,174 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 63.74 2021-05-03 02:48:33,345 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7234 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-7339 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:35,579 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8958 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9005 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:35,700 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7395 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:37,316 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 77.96 2021-05-03 02:48:37,676 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9179 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-9142 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9228 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-9148 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:48:37,677 | test | ERROR | pool-7-thread-9 | [task:call:3825] Failed to load 21 docs from 7532 to 9230 2021-05-03 02:48:37,678 | test | ERROR | pool-7-thread-9 | [task:call:3825] Failed to load 17 docs from 5763 to 7532 2021-05-03 02:48:37,678 | test | ERROR | pool-7-thread-9 | [task:call:3825] Failed to load 6 docs from 14610 to 16025 2021-05-03 02:48:38,125 | test | INFO | pool-7-thread-8 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................*..................................................**..............................X | 344.518 | | 315.752 | ..........................................................................................X | 315.752 | | 192.352 | ......................................................X | 192.352 | | 127.035 | ...................................*X | 127.035 | | 116.522 | ................................X | 116.522 | | 110.622 | ...............................*...X | 121.000 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:48:42,470 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 92.85 2021-05-03 02:48:47,619 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:48:52,687 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:49:02,841 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x175592 type=client open_count=1 channel=[id: 0xe0fd23c8, 0.0.0.0/0.0.0.0:37621] timeout=300.0>) 2021-05-03 02:49:03,084 | test | ERROR | pool-7-thread-10 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:49:05,234 | test | INFO | pool-7-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 59.0249998569 sec 2021-05-03 02:49:05,239 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 2 2021-05-03 02:49:05,329 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:49:05,339 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:49:05,359 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:49:07,394 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:49:09,467 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:49:11,513 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:49:13,293 | test | INFO | pool-7-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 7.36708860759 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 5.96562184024 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 10.8601877696 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:49:13,318 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:49:13,720 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:49:18,382 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:49:23,578 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 3.59 2021-05-03 02:49:27,736 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8563 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-8574 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8646 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-8620 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:27,813 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10424 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10356 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10322 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10398 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:28,796 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 19.99 2021-05-03 02:49:31,835 | test | INFO | pool-7-thread-19 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-17231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-17339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:31,836 | test | INFO | pool-7-thread-19 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-17231 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-17339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:32,891 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8793 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-8778 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-8796 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:33,933 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 40.01 2021-05-03 02:49:34,532 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:36,046 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9056 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9061 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8965 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9017 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-8952 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:37,950 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10676 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10732 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10777 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:38,071 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9157 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9183 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9171 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9169 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-9224 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:38,760 | test | ERROR | pool-7-thread-16 | [task:call:3825] Failed to load 12 docs from 9230 to 10860 2021-05-03 02:49:38,762 | test | ERROR | pool-7-thread-16 | [task:call:3825] Failed to load 18 docs from 7532 to 9230 2021-05-03 02:49:38,763 | test | ERROR | pool-7-thread-16 | [task:call:3825] Failed to load 2 docs from 16025 to 17384 2021-05-03 02:49:39,019 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 54.13 2021-05-03 02:49:39,984 | test | INFO | pool-7-thread-20 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***...........................................................***.........................................X | 344.300 | | 309.200 | ........................................................................................X | 309.200 | | 252.800 | ........................................................................*X | 252.800 | | 150.600 | ..........................................X | 150.600 | | 120.800 | ..................................*X | 120.800 | | 99.500 | ...........................X | 99.500 | | 79.300 | ......................X | 79.300 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:49:44,171 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 71.04 2021-05-03 02:49:49,288 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 88.68 2021-05-03 02:49:59,453 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:49:59,506 | test | INFO | pool-7-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 46.2120001316 sec 2021-05-03 02:49:59,562 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:49:59,569 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:49:59,604 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:49:59,709 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:59,733 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:59,742 | test | INFO | pool-7-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 48.5089974293 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 48.8935721812 | Cluster node | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 23.2170744404 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 62.2140415461 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:49:59,749 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:49:59,750 | test | ERROR | pool-7-thread-17 | [task:call:3825] Failed to load 4 docs from 10860 to 12425 2021-05-03 02:49:59,772 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:50:02,094 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11234 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:03,263 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:50:03,500 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9806 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9822 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-9658 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:04,875 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 6.0 2021-05-03 02:50:04,878 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11275 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11267 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:08,983 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10001 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9846 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:09,979 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 20.56 2021-05-03 02:50:12,085 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11495 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11567 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11505 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:14,223 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-18025 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:14,224 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18153 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-18025 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:15,118 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 35.42 2021-05-03 02:50:19,250 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10049 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10179 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10176 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:20,273 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 49.7 2021-05-03 02:50:21,799 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:23,173 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:25,381 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 62.18 2021-05-03 02:50:25,480 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11877 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:25,674 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18464 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:25,676 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18464 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:27,430 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10572 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10536 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10503 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10493 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10537 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:29,118 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10694 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10704 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:29,894 | test | ERROR | pool-7-thread-17 | [task:call:3825] Failed to load 11 docs from 10860 to 12425 2021-05-03 02:50:30,035 | test | ERROR | pool-7-thread-17 | [task:call:3825] Failed to load 18 docs from 9230 to 10860 2021-05-03 02:50:30,036 | test | ERROR | pool-7-thread-17 | [task:call:3825] Failed to load 3 docs from 17384 to 18688 2021-05-03 02:50:30,469 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 77.38 2021-05-03 02:50:31,062 | test | INFO | pool-7-thread-27 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.......................................**............................**.................................X | 312.900 | | 265.900 | ...................................................................................*X | 265.900 | | 199.600 | ..............................................................X | 199.600 | | 113.594 | ...................................X | 113.594 | | 64.294 | ...................**.X | 68.194 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:50:35,581 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 93.07 2021-05-03 02:50:40,677 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:50:45,755 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:50:55,910 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:50:56,403 | test | INFO | pool-7-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 56.6610000134 sec 2021-05-03 02:50:56,404 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 3 2021-05-03 02:50:56,450 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:50:56,454 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:50:56,556 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:50:56,584 | test | INFO | pool-7-thread-7 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:50:56,801 | test | ERROR | pool-7-thread-24 | [task:call:3825] Failed to load 1 docs from 12425 to 13928 2021-05-03 02:50:58,604 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:51:00,720 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:51:02,756 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:51:04,644 | test | INFO | pool-7-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 16.2985529322 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 27.5639369672 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 8.42824601367 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:51:04,727 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:51:05,207 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:51:09,842 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:51:15,122 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 5.13 2021-05-03 02:51:15,904 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12365 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12417 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:51:16,736 | test | INFO | pool-7-thread-2 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13891 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-13901 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:51:16,739 | test | ERROR | pool-7-thread-24 | [task:call:3825] Failed to load 2 docs from 12425 to 13928 2021-05-03 02:51:16,739 | test | ERROR | pool-7-thread-24 | [task:call:3825] Failed to load 2 docs from 10860 to 12425 2021-05-03 02:51:17,167 | test | INFO | pool-7-thread-4 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..........................*..............................*...........................................*X | 362.575 | | 346.575 | ..............................................................................................*X | 346.575 | | 299.375 | .................................................................................X | 299.375 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:51:20,279 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 20.18 2021-05-03 02:51:25,388 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 37.42 2021-05-03 02:51:30,562 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 54.3 2021-05-03 02:51:35,717 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.11 2021-05-03 02:51:40,938 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 88.33 2021-05-03 02:51:51,072 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:51:51,114 | test | INFO | pool-7-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 46.4679999352 sec 2021-05-03 02:51:51,171 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:51:51,177 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:51:51,256 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 02:51:51,358 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14399 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-14446 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:51:51,375 | test | INFO | pool-7-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 69.3360888659 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 58.3112758073 | Cluster node | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 26.1199895206 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 60.9325605901 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:51:51,440 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:51:51,459 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15141 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15136 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:51:51,469 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 4 docs from 13928 to 15370 2021-05-03 02:51:55,214 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 02:51:55,716 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14229 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:51:56,301 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-20260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:51:56,302 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20174 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-20260 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:51:56,700 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 6.4 2021-05-03 02:52:01,601 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12693 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12766 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12703 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:01,865 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 20.95 2021-05-03 02:52:02,867 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20530 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-20458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-20448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:02,868 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20530 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-20458 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-20448 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:04,941 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14573 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14632 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14539 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14637 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:05,986 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12870 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12995 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:06,891 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20563 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:06,891 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20563 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:07,000 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 34.58 2021-05-03 02:52:12,150 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 48.55 2021-05-03 02:52:12,226 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13224 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:14,641 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15062 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15027 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14966 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14929 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:16,993 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13352 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13420 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13248 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:17,371 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 62.16 2021-05-03 02:52:18,835 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-21021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-21038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:18,835 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-21021 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-21038 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:18,897 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15281 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15311 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15226 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:20,417 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15366 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15333 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:20,418 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 16 docs from 13928 to 15370 2021-05-03 02:52:22,484 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 77.66 2021-05-03 02:52:22,851 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in olZHb-14-553000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13826 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 02:52:22,851 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 10 docs from 12425 to 13928 2021-05-03 02:52:22,852 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 8 docs from 19940 to 21142 2021-05-03 02:52:22,986 | test | INFO | pool-7-thread-11 | [table_view:display:72] Ops trend for bucket 'olZHb-14-553000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **......................**..........................*....................................................X | 296.900 | | 259.100 | ......................................................................................*X | 259.100 | | 220.300 | .........................................................................X | 220.300 | | 121.490 | .......................................**...X | 129.290 | | 117.590 | ......................................X | 117.590 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:52:27,634 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 90.74 2021-05-03 02:52:32,740 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:52:37,819 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 02:52:47,927 | test | INFO | pool-7-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x1760ca type=client open_count=1 channel=[id: 0x217a15d3, 0.0.0.0/0.0.0.0:59242] timeout=300.0>) 2021-05-03 02:52:48,148 | test | ERROR | pool-7-thread-1 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:52:50,217 | test | INFO | pool-7-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 58.8410000801 sec 2021-05-03 02:52:51,457 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: olZHb-14-553000, Active Resident ratio(DGM): 100% 2021-05-03 02:52:51,457 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: olZHb-14-553000, Replica Resident ratio(DGM): 100% 2021-05-03 02:53:12,421 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 02:53:26,336 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2049999237 seconds to complete 2021-05-03 02:53:26,354 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete During the test, ok ---------------------------------------------------------------------- Ran 1 test in 759.128s OK Remote Connections: 62, Disconnections: 62 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_5 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,graceful=False' Test Input params: {'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'graceful': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_6', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 6, 'bulk_api_crud': 'True'} test_Orchestrator_Node_failover (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 02:53:36,263 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 02:53:37,230 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 02:53:37,776 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760cb type=client open_count=1 channel=[id: 0x367bf298, 0.0.0.0/0.0.0.0:57082] timeout=300.0>) 2021-05-03 02:53:43,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760cc type=client open_count=1 channel=[id: 0xd2df049a, 0.0.0.0/0.0.0.0:56887] timeout=300.0>) 2021-05-03 02:53:45,545 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760cd type=client open_count=1 channel=[id: 0x42329dec, 0.0.0.0/0.0.0.0:46471] timeout=300.0>) 2021-05-03 02:53:47,581 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760ce type=client open_count=1 channel=[id: 0x25bd4356, 0.0.0.0/0.0.0.0:52652] timeout=300.0>) 2021-05-03 02:53:49,605 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760cf type=client open_count=1 channel=[id: 0x9183f526, 0.0.0.0/0.0.0.0:44329] timeout=300.0>) 2021-05-03 02:53:51,624 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760d0 type=client open_count=1 channel=[id: 0x6fe70068, 0.0.0.0/0.0.0.0:51159] timeout=300.0>) 2021-05-03 02:53:53,642 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760d1 type=client open_count=1 channel=[id: 0x3a2c29d0, 0.0.0.0/0.0.0.0:60588] timeout=300.0>) 2021-05-03 02:53:55,691 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760d2 type=client open_count=1 channel=[id: 0x10c3629b, 0.0.0.0/0.0.0.0:42569] timeout=300.0>) 2021-05-03 02:53:57,845 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1760d3 type=client open_count=1 channel=[id: 0x6051a4e7, 0.0.0.0/0.0.0.0:40841] timeout=300.0>) 2021-05-03 02:53:59,895 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760d4 type=client open_count=1 channel=[id: 0x44ec8ae8, 0.0.0.0/0.0.0.0:47221] timeout=300.0>) 2021-05-03 02:54:08,388 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760d5 type=client open_count=1 channel=[id: 0xf7e7796b, 0.0.0.0/0.0.0.0:35562] timeout=300.0>) 2021-05-03 02:54:10,411 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760d6 type=client open_count=1 channel=[id: 0xfc8821c4, 0.0.0.0/0.0.0.0:46527] timeout=300.0>) 2021-05-03 02:54:12,451 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760d7 type=client open_count=1 channel=[id: 0xfd1eea50, 0.0.0.0/0.0.0.0:36357] timeout=300.0>) 2021-05-03 02:54:14,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760d8 type=client open_count=1 channel=[id: 0xd5fb305c, 0.0.0.0/0.0.0.0:38207] timeout=300.0>) 2021-05-03 02:54:16,513 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760d9 type=client open_count=1 channel=[id: 0x1e545368, 0.0.0.0/0.0.0.0:53461] timeout=300.0>) 2021-05-03 02:54:18,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760da type=client open_count=1 channel=[id: 0xc49c3bd3, 0.0.0.0/0.0.0.0:48557] timeout=300.0>) 2021-05-03 02:54:20,584 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760db type=client open_count=1 channel=[id: 0x66c71ca6, 0.0.0.0/0.0.0.0:59490] timeout=300.0>) 2021-05-03 02:54:22,634 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1760dc type=client open_count=1 channel=[id: 0x8857e98b, 0.0.0.0/0.0.0.0:42159] timeout=300.0>) 2021-05-03 02:54:24,661 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760dd type=client open_count=1 channel=[id: 0x35edb20a, 0.0.0.0/0.0.0.0:36313] timeout=300.0>) 2021-05-03 02:54:33,052 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760de type=client open_count=1 channel=[id: 0xdab62f12, 0.0.0.0/0.0.0.0:35309] timeout=300.0>) 2021-05-03 02:54:35,075 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760df type=client open_count=1 channel=[id: 0xef56ccf6, 0.0.0.0/0.0.0.0:60221] timeout=300.0>) 2021-05-03 02:54:37,108 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760e0 type=client open_count=1 channel=[id: 0x18a2301e, 0.0.0.0/0.0.0.0:60587] timeout=300.0>) 2021-05-03 02:54:39,128 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760e1 type=client open_count=1 channel=[id: 0x4edc8b9d, 0.0.0.0/0.0.0.0:56563] timeout=300.0>) 2021-05-03 02:54:41,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760e2 type=client open_count=1 channel=[id: 0x14e7b0a6, 0.0.0.0/0.0.0.0:46787] timeout=300.0>) 2021-05-03 02:54:43,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760e3 type=client open_count=1 channel=[id: 0x157c1810, 0.0.0.0/0.0.0.0:42291] timeout=300.0>) 2021-05-03 02:54:45,223 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760e4 type=client open_count=1 channel=[id: 0x0658fd36, 0.0.0.0/0.0.0.0:55601] timeout=300.0>) 2021-05-03 02:54:47,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760e5 type=client open_count=1 channel=[id: 0x2857e6a9, 0.0.0.0/0.0.0.0:37201] timeout=300.0>) 2021-05-03 02:54:49,273 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1760e6 type=client open_count=1 channel=[id: 0x99f2e08d, 0.0.0.0/0.0.0.0:52003] timeout=300.0>) 2021-05-03 02:54:51,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760e7 type=client open_count=1 channel=[id: 0xfb7b1fe8, 0.0.0.0/0.0.0.0:57779] timeout=300.0>) 2021-05-03 02:55:00,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760e8 type=client open_count=1 channel=[id: 0x29bc519b, 0.0.0.0/0.0.0.0:55195] timeout=300.0>) 2021-05-03 02:55:02,052 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760e9 type=client open_count=1 channel=[id: 0x4186a34c, 0.0.0.0/0.0.0.0:53957] timeout=300.0>) 2021-05-03 02:55:04,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760ea type=client open_count=1 channel=[id: 0xe07b7d54, 0.0.0.0/0.0.0.0:60269] timeout=300.0>) 2021-05-03 02:55:06,107 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760eb type=client open_count=1 channel=[id: 0x0ee1534b, 0.0.0.0/0.0.0.0:43127] timeout=300.0>) 2021-05-03 02:55:08,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760ec type=client open_count=1 channel=[id: 0xb4ac358d, 0.0.0.0/0.0.0.0:43575] timeout=300.0>) 2021-05-03 02:55:10,161 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760ed type=client open_count=1 channel=[id: 0xb35c9f41, 0.0.0.0/0.0.0.0:39629] timeout=300.0>) 2021-05-03 02:55:12,184 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760ee type=client open_count=1 channel=[id: 0x2ffa48ef, 0.0.0.0/0.0.0.0:53245] timeout=300.0>) 2021-05-03 02:55:14,217 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1760ef type=client open_count=1 channel=[id: 0x899a5ae8, 0.0.0.0/0.0.0.0:42569] timeout=300.0>) 2021-05-03 02:55:16,255 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x1760f0 type=client open_count=1 channel=[id: 0xbe4280c6, 0.0.0.0/0.0.0.0:46039] timeout=300.0>) 2021-05-03 02:55:21,441 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x1760f1 type=client open_count=1 channel=[id: 0x332b587f, 0.0.0.0/0.0.0.0:57242] timeout=300.0>) 2021-05-03 02:55:23,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x1760f2 type=client open_count=1 channel=[id: 0x10ffed9a, 0.0.0.0/0.0.0.0:51519] timeout=300.0>) 2021-05-03 02:55:25,507 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x1760f3 type=client open_count=1 channel=[id: 0x7f92a470, 0.0.0.0/0.0.0.0:55693] timeout=300.0>) 2021-05-03 02:55:27,542 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x1760f4 type=client open_count=1 channel=[id: 0x6c5e49ab, 0.0.0.0/0.0.0.0:52203] timeout=300.0>) 2021-05-03 02:55:29,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x1760f5 type=client open_count=1 channel=[id: 0x30972eec, 0.0.0.0/0.0.0.0:40711] timeout=300.0>) 2021-05-03 02:55:31,604 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 02:55:48,724 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 02:56:02,944 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 02:56:02,973 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 02:56:03,019 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 02:56:34,918 | test | INFO | pool-8-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 12.8424227193 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:56:34,957 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:56:45,099 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:56:45,132 | test | INFO | pool-8-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2139999866 sec 2021-05-03 02:56:45,617 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 02:56:45,986 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 02:56:46,260 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | UVMnNJcX1n-45-723000 | couchbase | 2 | none | 0 | 0 | 10825498624 | 0 | 0 | +----------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-03 02:56:46,262 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 02:56:49,289 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 02:56:49,296 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 02:57:19,674 | test | INFO | pool-8-thread-9 | [table_view:display:72] Ops trend for bucket 'UVMnNJcX1n-45-723000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 75.100 | ...................*...............................................**..........................***.......*X | 365.800 | | 360.800 | .................................................................................................*X | 360.800 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 02:57:19,742 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 21.210540047 | 3693387776 | 2971623424 | 0 / 3909087232 | 1651 / 3323 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 20.1459473547 | 3693387776 | 3070320640 | 0 / 3909087232 | 2149 / 4345 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 20.6958301356 | 6078930944 | 5239316480 | 0 / 3758092288 | 1801 / 3547 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 26.2788886004 | 4201836544 | 3579162624 | 0 / 3758092288 | 1808 / 3672 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 02:57:27,019 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 02:57:54,141 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +----------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +----------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | UVMnNJcX1n-45-723000 | couchbase | 2 | none | 0 | 10000 | 10825498624 | 103764768 | 183356164 | +----------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 02:57:54,142 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 02:58:08,040 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 02:58:30,246 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 0 2021-05-03 02:58:39,835 | test | INFO | pool-8-thread-23 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.09699988365 seconds to complete 2021-05-03 02:58:39,842 | test | INFO | pool-8-thread-23 | [common_lib:sleep:22] Sleep 8.09699988365 seconds. Reason: Wait after rebalance complete 2021-05-03 02:58:48,141 | test | INFO | pool-8-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 1.65537998495 | --- OUT ---> | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 2.47349823322 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 4.99747602221 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 3.43607882769 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:58:48,164 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:58:53,257 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 15.76 2021-05-03 02:58:58,349 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 39.07 2021-05-03 02:59:03,480 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 62.77 2021-05-03 02:59:08,586 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 86.73 Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1761f6 type=client open_count=1 channel=[id: 0x034c9c9c, 0.0.0.0/0.0.0.0:43091] timeout=300.0>) 2021-05-03 02:59:13,631 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1761f7 type=client open_count=1 channel=[id: 0x5f192ce6, 0.0.0.0/0.0.0.0:46673] timeout=300.0>) 2021-05-03 02:59:15,641 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1761f8 type=client open_count=1 channel=[id: 0xc67784ea, 0.0.0.0/0.0.0.0:37585] timeout=300.0>) 2021-05-03 02:59:17,654 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1761f9 type=client open_count=1 channel=[id: 0xf778affb, 0.0.0.0/0.0.0.0:49081] timeout=300.0>) 2021-05-03 02:59:19,665 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x1761fa type=client open_count=1 channel=[id: 0x685b0ec0, 0.0.0.0/0.0.0.0:56015] timeout=300.0>) 2021-05-03 02:59:21,677 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused 2021-05-03 02:59:28,750 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 02:59:28,773 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:748] GET http://172.23.98.196:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.98.196:8091/pools/default with status 0: unknown pool 2021-05-03 02:59:28,851 | test | INFO | pool-8-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 40.7100000381 sec 2021-05-03 02:59:28,967 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.104.186 2021-05-03 02:59:28,967 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.98.196 2021-05-03 02:59:40,864 | test | INFO | pool-8-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.56420626896 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 4.9684741488 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 4.89350912779 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 02:59:40,892 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:59:46,006 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 02:59:51,091 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 8.2 2021-05-03 02:59:56,207 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-03 03:00:01,313 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 52.3 2021-05-03 03:00:06,470 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 73.44 2021-05-03 03:00:11,569 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 94.8 2021-05-03 03:00:21,704 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 03:00:21,737 | test | INFO | pool-8-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 40.875 sec 2021-05-03 03:00:21,737 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 1 2021-05-03 03:00:31,500 | test | INFO | pool-8-thread-26 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.12599992752 seconds to complete 2021-05-03 03:00:31,506 | test | INFO | pool-8-thread-26 | [common_lib:sleep:22] Sleep 8.12599992752 seconds. Reason: Wait after rebalance complete 2021-05-03 03:00:39,990 | test | INFO | pool-8-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 2.61898766054 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.48925410872 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 6.7946451124 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 4.35553304634 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:00:40,040 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:00:45,128 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 16.59 2021-05-03 03:00:50,255 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 38.61 2021-05-03 03:00:55,421 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 63.29 2021-05-03 03:01:00,532 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 87.27 2021-05-03 03:01:10,631 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1761fb type=client open_count=1 channel=[id: 0x07f99f07, 0.0.0.0/0.0.0.0:37915] timeout=300.0>) 2021-05-03 03:01:10,671 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1761fc type=client open_count=1 channel=[id: 0x79df0bb9, 0.0.0.0/0.0.0.0:59599] timeout=300.0>) 2021-05-03 03:01:12,697 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1761fd type=client open_count=1 channel=[id: 0x29b4fbd1, 0.0.0.0/0.0.0.0:33486] timeout=300.0>) 2021-05-03 03:01:14,709 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 03:01:16,786 | test | INFO | pool-8-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 36.7969999313 sec 2021-05-03 03:01:16,931 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.121.10 2021-05-03 03:01:16,933 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.104.186 2021-05-03 03:01:27,536 | test | INFO | pool-8-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 4.21185372005 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 5.66801619433 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.04154002026 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:01:27,562 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:01:32,654 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:01:37,773 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:01:42,874 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 8.95 2021-05-03 03:01:47,976 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 25.63 2021-05-03 03:01:53,063 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 44.18 2021-05-03 03:01:58,270 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 63.63 2021-05-03 03:02:03,417 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 86.37 2021-05-03 03:02:13,536 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 03:02:13,572 | test | INFO | pool-8-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 46.0379998684 sec 2021-05-03 03:02:13,573 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 2 2021-05-03 03:02:23,096 | test | INFO | pool-8-thread-29 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.15299987793 seconds to complete 2021-05-03 03:02:23,102 | test | INFO | pool-8-thread-29 | [common_lib:sleep:22] Sleep 8.15299987793 seconds. Reason: Wait after rebalance complete 2021-05-03 03:02:31,474 | test | INFO | pool-8-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 2.99672626542 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 7.8720162519 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 7.34662963898 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 8.71548011148 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:02:31,513 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:02:36,664 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 13.85 2021-05-03 03:02:41,766 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 33.48 2021-05-03 03:02:46,839 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 55.16 2021-05-03 03:02:51,930 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 83.42 2021-05-03 03:02:57,194 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 03:03:07,384 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1761fe type=client open_count=1 channel=[id: 0x474a9e22, 0.0.0.0/0.0.0.0:53231] timeout=300.0>) 2021-05-03 03:03:07,529 | test | ERROR | pool-8-thread-30 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1761ff type=client open_count=1 channel=[id: 0x1f8ad1cf, 0.0.0.0/0.0.0.0:59708] timeout=300.0>) 2021-05-03 03:03:09,553 | test | ERROR | pool-8-thread-30 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 03:03:11,888 | test | INFO | pool-8-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 40.4130001068 sec 2021-05-03 03:03:12,131 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.104.186 2021-05-03 03:03:12,132 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.121.10 2021-05-03 03:03:25,216 | test | INFO | pool-8-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 5.55275113579 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 6.07594936709 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 4.51450189155 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:03:25,242 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:03:30,305 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:03:35,368 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:03:40,528 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 8.34 2021-05-03 03:03:45,730 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.47 2021-05-03 03:03:50,816 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 48.91 2021-05-03 03:03:55,903 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.19 2021-05-03 03:04:01,025 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 89.2 2021-05-03 03:04:11,240 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 03:04:11,319 | test | INFO | pool-8-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 46.1040000916 sec 2021-05-03 03:04:11,321 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 3 2021-05-03 03:04:21,236 | test | INFO | pool-8-thread-4 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.11999988556 seconds to complete 2021-05-03 03:04:21,243 | test | INFO | pool-8-thread-4 | [common_lib:sleep:22] Sleep 8.11999988556 seconds. Reason: Wait after rebalance complete 2021-05-03 03:04:29,572 | test | INFO | pool-8-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 3.02190883908 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.05324249306 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 7.42424242424 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 3.38383838384 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:04:29,601 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:04:34,693 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 18.98 2021-05-03 03:04:39,841 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 45.16 2021-05-03 03:04:44,937 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.1 2021-05-03 03:04:50,052 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: running, progress: 97.7 2021-05-03 03:05:00,157 | test | INFO | pool-8-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x176200 type=client open_count=1 channel=[id: 0x2f1fd804, 0.0.0.0/0.0.0.0:44131] timeout=300.0>) 2021-05-03 03:05:00,204 | test | ERROR | pool-8-thread-1 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 03:05:02,486 | test | INFO | pool-8-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 32.9149999619 sec 2021-05-03 03:05:02,549 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-03 03:05:03,638 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.121.10 2021-05-03 03:05:03,638 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.104.186 2021-05-03 03:05:14,460 | test | INFO | pool-8-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.26420813337 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.64280293448 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 5.93306288032 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:05:14,591 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:05:19,673 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:05:24,746 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:05:29,871 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 12.03 2021-05-03 03:05:34,996 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 31.68 2021-05-03 03:05:40,153 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 52.04 2021-05-03 03:05:45,349 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 73.36 2021-05-03 03:05:50,535 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: running, progress: 94.22 2021-05-03 03:06:00,710 | test | INFO | pool-8-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 03:06:00,816 | test | INFO | pool-8-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 46.3579998016 sec 2021-05-03 03:06:02,213 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: UVMnNJcX1n-45-723000, Active Resident ratio(DGM): 100% 2021-05-03 03:06:02,213 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: UVMnNJcX1n-45-723000, Replica Resident ratio(DGM): 100% 2021-05-03 03:06:25,094 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 03:06:39,177 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2739999294 seconds to complete 2021-05-03 03:06:39,194 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 793.315s OK During the test, Remote Connections: 82, Disconnections: 82 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 6 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_00-17-11/test_6 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=3,nodes_swap=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=3,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 2h 49m 47s 2 actionable tasks: 1 executed, 1 up-to-date + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_rebalance_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'cluster_name': 'win10-bucket-ops', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops.ini', 'num_nodes': 5, 'quota_percent': '95', 'rerun': 'False', 'spec': 'collections_rebalance_crud_on_collections'} Only cases in GROUPs 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' will be executed Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_1', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 1, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 03:07:12,390 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 03:07:13,822 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 03:07:15,687 | test | ERROR | MainThread | [rest_client:_http_request:748] GET http://172.23.98.196:8091/pools/default/ body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-05-03 03:07:16,821 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xee77fba1, 0.0.0.0/0.0.0.0:47517] timeout=300.0>) 2021-05-03 03:07:22,660 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x495c37d4, 0.0.0.0/0.0.0.0:47599] timeout=300.0>) 2021-05-03 03:07:25,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x62122a22, 0.0.0.0/0.0.0.0:56025] timeout=300.0>) 2021-05-03 03:07:27,980 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xa56b81e0, 0.0.0.0/0.0.0.0:47993] timeout=300.0>) 2021-05-03 03:07:30,065 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xa5d1f888, 0.0.0.0/0.0.0.0:45819] timeout=300.0>) 2021-05-03 03:07:32,135 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xbe1637f5, 0.0.0.0/0.0.0.0:52373] timeout=300.0>) 2021-05-03 03:07:34,178 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x8786facf, 0.0.0.0/0.0.0.0:50023] timeout=300.0>) 2021-05-03 03:07:36,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0xcb4d4e77, 0.0.0.0/0.0.0.0:55700] timeout=300.0>) 2021-05-03 03:07:38,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x3b424dcc, 0.0.0.0/0.0.0.0:44641] timeout=300.0>) 2021-05-03 03:07:46,032 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0xa2b2741b, 0.0.0.0/0.0.0.0:42042] timeout=300.0>) 2021-05-03 03:07:48,075 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0xd760119a, 0.0.0.0/0.0.0.0:56861] timeout=300.0>) 2021-05-03 03:07:50,112 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x204bca49, 0.0.0.0/0.0.0.0:58998] timeout=300.0>) 2021-05-03 03:07:52,203 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x659fd8c6, 0.0.0.0/0.0.0.0:50195] timeout=300.0>) 2021-05-03 03:07:54,265 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x4268cb67, 0.0.0.0/0.0.0.0:37523] timeout=300.0>) 2021-05-03 03:07:56,296 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x11da674c, 0.0.0.0/0.0.0.0:48061] timeout=300.0>) 2021-05-03 03:07:58,329 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0xadeb31f8, 0.0.0.0/0.0.0.0:56299] timeout=300.0>) 2021-05-03 03:08:00,361 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0xb75bf665, 0.0.0.0/0.0.0.0:55649] timeout=300.0>) 2021-05-03 03:08:02,394 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0x1573d970, 0.0.0.0/0.0.0.0:58713] timeout=300.0>) 2021-05-03 03:08:10,098 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x3f2a205a, 0.0.0.0/0.0.0.0:59567] timeout=300.0>) 2021-05-03 03:08:12,132 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0xb1e33408, 0.0.0.0/0.0.0.0:50151] timeout=300.0>) 2021-05-03 03:08:14,168 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0x971be29d, 0.0.0.0/0.0.0.0:56147] timeout=300.0>) 2021-05-03 03:08:16,210 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x5b3a03a0, 0.0.0.0/0.0.0.0:41549] timeout=300.0>) 2021-05-03 03:08:18,247 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0xd62fb895, 0.0.0.0/0.0.0.0:56476] timeout=300.0>) 2021-05-03 03:08:20,282 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x2c52d76e, 0.0.0.0/0.0.0.0:33603] timeout=300.0>) 2021-05-03 03:08:22,313 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0x8b9bf8f8, 0.0.0.0/0.0.0.0:41769] timeout=300.0>) 2021-05-03 03:08:24,361 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0x902fd5bf, 0.0.0.0/0.0.0.0:45513] timeout=300.0>) 2021-05-03 03:08:26,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0x68d2e5ee, 0.0.0.0/0.0.0.0:48925] timeout=300.0>) 2021-05-03 03:08:28,490 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0x6bb5c3f1, 0.0.0.0/0.0.0.0:33102] timeout=300.0>) 2021-05-03 03:08:37,770 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0xbca60c71, 0.0.0.0/0.0.0.0:59440] timeout=300.0>) 2021-05-03 03:08:39,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2a type=client open_count=1 channel=[id: 0x495f0416, 0.0.0.0/0.0.0.0:44681] timeout=300.0>) 2021-05-03 03:08:41,842 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2b type=client open_count=1 channel=[id: 0x7349bcd0, 0.0.0.0/0.0.0.0:54765] timeout=300.0>) 2021-05-03 03:08:43,969 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2c type=client open_count=1 channel=[id: 0x5eb47531, 0.0.0.0/0.0.0.0:46391] timeout=300.0>) 2021-05-03 03:08:46,073 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2d type=client open_count=1 channel=[id: 0xe9087ae3, 0.0.0.0/0.0.0.0:45249] timeout=300.0>) 2021-05-03 03:08:48,164 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2e type=client open_count=1 channel=[id: 0x14b8af8d, 0.0.0.0/0.0.0.0:39177] timeout=300.0>) 2021-05-03 03:08:50,301 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2f type=client open_count=1 channel=[id: 0x52a31813, 0.0.0.0/0.0.0.0:39329] timeout=300.0>) 2021-05-03 03:08:52,381 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x30 type=client open_count=1 channel=[id: 0x9369245f, 0.0.0.0/0.0.0.0:38009] timeout=300.0>) 2021-05-03 03:08:54,430 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x31 type=client open_count=1 channel=[id: 0xcb6858f3, 0.0.0.0/0.0.0.0:60667] timeout=300.0>) 2021-05-03 03:08:59,990 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x32 type=client open_count=1 channel=[id: 0x47147a14, 0.0.0.0/0.0.0.0:55926] timeout=300.0>) 2021-05-03 03:09:02,023 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x33 type=client open_count=1 channel=[id: 0x6b4eb414, 0.0.0.0/0.0.0.0:58353] timeout=300.0>) 2021-05-03 03:09:04,068 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x34 type=client open_count=1 channel=[id: 0x86e2fb26, 0.0.0.0/0.0.0.0:44961] timeout=300.0>) 2021-05-03 03:09:06,117 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x35 type=client open_count=1 channel=[id: 0x68b5e2e3, 0.0.0.0/0.0.0.0:44698] timeout=300.0>) 2021-05-03 03:09:08,170 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x36 type=client open_count=1 channel=[id: 0xb344ef6b, 0.0.0.0/0.0.0.0:52485] timeout=300.0>) 2021-05-03 03:09:10,193 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 03:09:29,124 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 03:09:44,332 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 03:09:44,362 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 03:09:44,417 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 03:10:07,417 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 13.0303030303 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:10:07,526 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:10:17,693 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 03:10:17,727 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.3199999332 sec 2021-05-03 03:10:18,141 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 03:11:05,151 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 03:11:05,536 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 629145600 | 673018808 | 48698928 | | bucket2 | couchbase | 2 | none | 0 | 0 | 629145600 | 668067128 | 63645649 | | default | couchbase | 2 | none | 0 | 0 | 6291456000 | 798585272 | 29390526 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-03 03:11:05,538 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 03:12:16,520 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 03:12:16,556 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 03:34:16,204 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****.*.*X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ********************************************................*...............*.............*X | 23.500 | | 15.500 | ............................*X | 15.500 | | 0.000 | *********.*..**..X | 3.000 | | 2.000 | ..X | 2.000 | | 1.000 | **...........**..............................................***.X | 31.515 | | 23.012 | ..........................................X | 23.012 | | 9.500 | .................*X | 9.500 | | 2.500 | ...*X | 2.500 | | 0.000 | *****.**.*X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ***************************************************************************************************..**.........**.......*....*X | 12.000 | | 7.500 | .............*X | 7.500 | | 3.500 | .....X | 3.500 | | 0.700 | **..........*.........................*..............................**...............................**...X | 52.613 | | 46.227 | ......................................................................................X | 46.227 | | 30.205 | ........................................................*X | 30.205 | | 20.705 | ......................................X | 20.705 | | 15.500 | ............................*X | 15.500 | | 10.000 | ..................*X | 10.000 | | 0.000 | ********.*.**.**..*X | 3.001 | | 2.001 | ..*X | 2.001 | | 1.000 | X | 1.000 | | 0.000 | **..**....*....*.........**..........*.........*X | 20.500 | | 15.000 | ...........................*X | 15.000 | | 8.000 | ..............X | 8.000 | | 1.000 | *X | 1.000 | | 0.000 | ***************************************************************************.*.*X | 1.500 | | 0.999 | X | 0.999 | | 0.500 | X | 0.500 | | 0.000 | *********************....**.....*.....X | 7.499 | | 4.998 | ........*X | 4.998 | | 2.498 | ...X | 2.498 | | 0.000 | ***************************.**.**..***.X | 3.000 | | 2.000 | ..X | 2.000 | | 1.000 | *X | 1.000 | | 0.000 | ************************************.*........**......*.....*X | 11.000 | | 7.000 | ............*X | 7.000 | | 0.000 | **************.***.X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | **.................***...............*......*.......*X | 24.005 | | 20.503 | .....................................*X | 20.503 | | 4.500 | .......*X | 4.500 | | 0.000 | *******.....**....*X | 5.000 | | 4.000 | ......*X | 4.000 | | 3.000 | ....***...*...*X | 6.001 | | 4.001 | ......*X | 4.001 | | 0.000 | ******.*..**..****X | 3.001 | | 2.001 | ..*X | 2.001 | | 1.000 | *X | 1.000 | | 0.000 | *********************************..*...*...*X | 4.500 | | 3.000 | ....X | 3.000 | | 1.500 | .*X | 1.500 | | 0.000 | ****************************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 03:34:17,960 | test | INFO | pool-2-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ********************************.**.**.X | 1.499 | | 0.999 | .*X | 0.999 | | 0.500 | **.................**.........*X | 11.997 | | 7.994 | .................X | 7.994 | | 3.997 | ........*X | 3.997 | | 0.000 | **.....*.............*..................**X | 15.600 | | 14.400 | ................................**X | 14.400 | | 7.400 | ................X | 7.400 | | 2.900 | .....X | 2.900 | | 1.500 | ..**..**.....**X | 4.500 | | 3.000 | .....X | 3.000 | | 1.500 | ..X | 1.500 | | 0.000 | **********************.*.*.*X | 1.500 | | 1.000 | .*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ************************************************************************************.*.*.*X | 1.499 | | 0.999 | .X | 0.999 | | 0.499 | **X | 0.499 | | 0.000 | ******..*.........**...............**........*X | 14.700 | | 8.200 | ..................*X | 8.200 | | 2.700 | .....X | 2.700 | | 0.000 | **********..........**......................*.......................**...*X | 25.000 | | 6.000 | ............*X | 6.000 | | 0.500 | *X | 0.500 | | 0.000 | **.........**...........................*.................................**...............................*X | 42.987 | | 41.010 | ..............................................................................................X | 41.010 | | 33.008 | ...........................................................................*X | 33.008 | | 16.008 | ....................................X | 16.008 | | 6.497 | ..............X | 6.497 | | 0.000 | **...........**...................*.................................**.....................**.......X | 39.400 | | 29.600 | ...................................................................X | 29.600 | | 15.600 | ...................................*X | 15.600 | | 4.600 | .........X | 4.600 | | 0.000 | ********************************************************************************************************..................*............................*..........................*X | 31.194 | | 28.189 | ................................................................*X | 28.189 | | 16.189 | ....................................X | 16.189 | | 4.500 | .........*X | 4.500 | | 0.000 | *****************************......**....*X | 4.501 | | 3.001 | .....*X | 3.001 | | 1.500 | ..X | 1.500 | | 0.000 | *****************.*..***..**X | 2.500 | | 1.500 | ..X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | **********.**..*..**X | 2.500 | | 1.500 | ..***.*....*X | 3.500 | | 3.000 | .....*X | 3.000 | | 2.500 | ....X | 2.500 | | 1.000 | .*X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ***********.**.....*......**...........**..*X | 11.002 | | 6.001 | ............*X | 6.001 | | 3.001 | .....X | 3.001 | | 0.000 | ********.*.**..*X | 2.000 | | 2.000 | ...**X | 2.000 | | 1.500 | ..****X | 1.500 | | 1.000 | .X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | *********...***...*....X | 4.502 | | 3.002 | .....*X | 3.002 | | 1.500 | ..*X | 1.500 | | 0.000 | ****....*.....**....*X | 5.998 | | 4.000 | ........X | 4.000 | | 2.000 | ...*X | 2.000 | | 0.000 | ******.*.**.*X | 1.500 | | 1.000 | .X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | *******.*.*.*X | 1.500 | | 1.000 | .X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | *****************************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 03:34:19,783 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..............**................................*..................................**....................X | 892.000 | | 847.400 | ..............................................................................................X | 847.400 | | 754.500 | ...................................................................................*X | 754.500 | | 656.000 | ........................................................................*X | 656.000 | | 611.900 | ...................................................................X | 611.900 | | 605.200 | ..................................................................X | 605.200 | | 587.049 | ................................................................*X | 587.049 | | 562.949 | ..............................................................X | 562.949 | | 539.649 | ...........................................................**.**.....*..*X | 613.720 | | 607.897 | ...................................................................*.*...**.....*X | 687.347 | | 595.894 | .................................................................**X | 595.894 | | 532.409 | ..........................................................*.*.*X | 551.947 | | 505.286 | .......................................................*X | 505.286 | | 366.798 | ........................................***X | 370.706 | | 353.862 | ......................................**X | 354.822 | | 343.997 | .....................................**..**X | 362.884 | | 340.500 | .....................................X | 340.500 | | 316.400 | ..................................X | 316.400 | | 297.800 | ................................**....**..*...X | 382.464 | | 378.999 | .........................................**..**.***..*....*...*X | 488.500 | | 458.261 | ..................................................X | 458.261 | | 410.655 | .............................................*X | 410.655 | | 355.155 | ......................................*X | 355.155 | | 353.649 | ......................................***...*...*...**...*X | 460.600 | | 425.500 | ..............................................X | 425.500 | | 385.478 | ..........................................X | 385.478 | | 340.778 | .....................................**...*....X | 407.230 | | 392.330 | ..........................................**.*X | 392.630 | | 354.400 | ......................................*...*X | 375.000 | | 324.100 | ...................................X | 324.100 | | 302.428 | ................................X | 302.428 | | 282.828 | ..............................**...**...*.*X | 344.596 | | 336.696 | ....................................*.*..**....**...*..**.....*..***X | 508.433 | | 441.700 | ................................................*X | 441.700 | | 406.400 | ............................................*X | 406.400 | | 385.800 | ..........................................X | 385.800 | | 362.200 | .......................................**....**....*........*X | 507.020 | | 494.273 | ......................................................*X | 494.273 | | 435.926 | ...............................................*X | 435.926 | | 423.453 | ..............................................*X | 423.453 | | 372.597 | ........................................X | 372.597 | | 315.397 | ..................................*X | 315.397 | | 271.497 | .............................**...**..**X | 320.600 | | 313.000 | ..................................X | 313.000 | | 303.661 | .................................*X | 303.661 | | 288.761 | ...............................X | 288.761 | | 284.961 | ..............................**.**....**.**..*.*.*X | 373.623 | | 371.723 | ........................................**..*..**...****X | 436.000 | | 407.100 | ............................................***.*...X | 438.455 | | 422.155 | ..............................................*X | 422.155 | | 413.031 | .............................................*X | 413.031 | | 402.931 | ............................................*....X | 442.131 | | 429.292 | ...............................................*X | 429.292 | | 400.913 | ...........................................X | 400.913 | | 350.313 | ......................................*X | 350.313 | | 325.821 | ...................................*X | 325.821 | | 323.477 | ...................................X | 323.477 | | 294.477 | ................................*X | 294.477 | | 277.177 | ..............................**.***...*..X | 334.000 | | 323.100 | ...................................*X | 323.100 | | 312.101 | .................................*X | 312.101 | | 252.163 | ...........................*X | 252.163 | | 246.055 | ..........................**....*.....*..*X | 339.460 | | 323.060 | ...................................X | 323.060 | | 293.200 | ...............................**.**........**...***....*..*X | 450.312 | | 428.079 | ..............................................X | 428.079 | | 373.167 | ........................................*X | 373.167 | | 333.767 | ....................................*X | 333.767 | | 332.090 | ....................................**....**..*.......*X | 447.565 | | 432.365 | ...............................................X | 432.365 | | 399.065 | ...........................................X | 399.065 | | 319.400 | ..................................*X | 319.400 | | 273.000 | .............................*X | 273.000 | | 228.100 | ........................*..**...**......*.X | 336.700 | | 325.400 | ...................................*X | 325.400 | | 294.800 | ................................***...**....X | 363.096 | | 362.896 | .......................................*X | 362.896 | | 354.696 | ......................................X | 354.696 | | 337.491 | ....................................*..**..**....*..**....**..X | 473.202 | | 471.502 | ...................................................X | 471.502 | | 462.302 | ..................................................*X | 462.302 | | 439.128 | ................................................*.***X | 450.828 | | 439.252 | ................................................X | 439.252 | | 404.352 | ............................................*X | 404.352 | | 350.552 | ......................................X | 350.552 | | 332.100 | ....................................X | 332.100 | | 316.000 | ..................................**.*X | 321.200 | | 320.200 | ..................................*...*..**..X | 383.063 | | 362.763 | .......................................*X | 362.763 | | 352.834 | ......................................*X | 352.834 | | 328.334 | ...................................X | 328.334 | | 321.534 | ...................................X | 321.534 | | 321.300 | ...................................**.*.....**..*.*.**.**.***..X | 451.200 | | 401.900 | ............................................X | 401.900 | | 390.700 | ..........................................*X | 390.700 | | 319.522 | ..................................*X | 319.522 | | 299.922 | ................................***.....X | 340.389 | | 336.332 | ....................................*.*X | 344.969 | | 329.303 | ...................................**...***X | 355.531 | | 333.764 | ....................................X | 333.764 | | 318.292 | ..................................*X | 318.292 | | 307.992 | .................................**....*...*..**X | 391.964 | | 383.739 | ..........................................**..*X | 408.331 | | 390.431 | ..........................................*....*X | 426.689 | | 395.600 | ...........................................*X | 395.600 | | 267.616 | .............................**X | 267.616 | | 255.548 | ...........................*..*.......**.*X | 347.021 | | 343.494 | .....................................*X | 344.551 | | 324.922 | ...................................***..X | 341.937 | | 314.575 | ..................................*...*X | 347.643 | | 334.943 | ....................................*....*X | 370.539 | | 367.100 | ........................................**.X | 380.900 | | 375.200 | .........................................X | 375.200 | | 368.500 | ........................................**..X | 383.612 | | 380.512 | .........................................*.*X | 385.250 | | 382.838 | .........................................**...*...*X | 430.015 | | 419.815 | ..............................................X | 419.815 | | 405.426 | ............................................X | 405.426 | | 378.453 | .........................................*X | 378.453 | | 346.105 | .....................................*X | 346.105 | | 296.053 | ................................X | 296.053 | | 267.453 | ............................*X | 267.453 | | 257.608 | ...........................*.*X | 265.308 | | 247.908 | ..........................**....*..*......*X | 350.852 | | 347.005 | .....................................*X | 347.005 | | 340.770 | .....................................X | 340.770 | | 334.490 | ....................................*X | 334.490 | | 311.890 | .................................*.*X | 317.058 | | 296.100 | ................................**.......*.....*.......**X | 470.951 | | 457.323 | ..................................................X | 457.323 | | 446.023 | .................................................*X | 446.023 | | 428.987 | ...............................................*.**X | 441.140 | | 428.852 | ...............................................*X | 428.852 | | 395.208 | ...........................................X | 395.208 | | 363.508 | .......................................*X | 363.508 | | 332.081 | ....................................*X | 332.081 | | 263.000 | ............................*X | 263.000 | | 248.500 | ..........................**.....**.....*...*X | 357.376 | | 341.276 | .....................................*.**..*X | 373.126 | | 364.800 | .......................................*X | 364.800 | | 353.651 | ......................................**.*.....**.*X | 418.225 | | 411.978 | .............................................X | 411.978 | | 388.178 | ..........................................*.**X | 397.498 | | 343.633 | .....................................*X | 343.633 | | 283.377 | ..............................**.*...**......**...*X | 394.700 | | 369.000 | ........................................*X | 369.000 | | 352.700 | ......................................X | 352.700 | | 335.500 | ....................................X | 335.500 | | 320.600 | ..................................*X | 320.600 | | 276.700 | ..............................*X | 276.700 | | 269.300 | .............................X | 269.300 | | 260.800 | ............................**.....***.*X | 317.553 | | 293.037 | ...............................*.***..*X | 317.679 | | 315.279 | ..................................**..*..*......**..**X | 425.860 | | 407.260 | ............................................X | 407.260 | | 397.030 | ...........................................*X | 397.030 | | 386.630 | ..........................................*.**.*X | 403.159 | | 393.559 | ...........................................X | 393.559 | | 347.127 | .....................................*X | 347.127 | | 298.027 | ................................*X | 298.027 | | 292.923 | ...............................**X | 292.923 | | 278.623 | ..............................*..X | 297.194 | | 286.694 | ...............................**......**...**X | 367.247 | | 355.424 | ......................................X | 355.424 | | 340.824 | .....................................*X | 340.824 | | 337.024 | ....................................X | 337.024 | | 328.923 | ...................................*X | 328.923 | | 323.623 | ...................................*..**.*....*..**..**..*.**.*X | 456.000 | | 395.100 | ...........................................*X | 395.100 | | 378.400 | .........................................**.X | 384.500 | | 351.700 | ......................................*X | 351.700 | | 325.869 | ...................................**X | 329.269 | | 326.969 | ...................................*..*X | 344.036 | | 317.236 | ..................................**.X | 324.736 | | 311.900 | .................................**...*.**.*X | 352.008 | | 346.608 | .....................................X | 346.608 | | 332.526 | ....................................X | 332.526 | | 330.672 | ....................................***..**.**.......*......*.........**...*...*X | 613.000 | | 607.100 | ...................................................................*X | 607.100 | | 546.600 | ............................................................X | 546.600 | | 463.800 | ..................................................*X | 463.800 | | 320.200 | ..................................X | 320.200 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 03:34:21,003 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 43.2382945331 | 3693387776 | 2421919744 | 0 / 3909087232 | 168800 / 337200 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 47.619047619 | 3693387776 | 2483261440 | 0 / 3909087232 | 168800 / 337196 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 47.8756476684 | 4201836544 | 2970681344 | 0 / 3758092288 | 168288 / 337398 | 7.0.0-5085-enterprise | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 03:34:34,349 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 03:36:10,183 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 153138104 | 257855768 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 153004936 | 269916297 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 429814136 | 441176840 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-03 03:36:10,186 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 03:36:17,780 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_in 2021-05-03 03:36:17,785 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_in 2021-05-03 03:36:17,895 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 03:36:25,036 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 03:36:37,275 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 19.7659628593 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 20.263091323 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 9.83481575604 | Cluster node | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:36:37,348 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:36:43,894 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:36:49,306 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 1.57 2021-05-03 03:36:54,500 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 5.93 2021-05-03 03:36:59,644 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 9.68 2021-05-03 03:37:04,857 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 13.15 2021-05-03 03:37:06,581 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:HhUXyBP56XIwcU8sHiBW-22-657000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-302 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-365 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-292 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:06,582 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:HhUXyBP56XIwcU8sHiBW-22-657000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-302 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-215 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-365 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-385 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-292 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:10,101 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 17.16 2021-05-03 03:37:15,278 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 20.4 2021-05-03 03:37:18,803 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:HhUXyBP56XIwcU8sHiBW-22-657000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-704 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-694 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-750 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:18,805 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:HhUXyBP56XIwcU8sHiBW-22-657000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-704 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-676 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-694 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-750 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:20,394 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 23.72 2021-05-03 03:37:23,813 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:aiFu8VpTTc%-22-658000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-111 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-82 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:23,815 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:aiFu8VpTTc%-22-658000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-111 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-82 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:25,516 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 27.04 2021-05-03 03:37:30,665 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 30.99 2021-05-03 03:37:33,128 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:aiFu8VpTTc%-22-658000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-303 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:33,130 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:aiFu8VpTTc%-22-658000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-303 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-293 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:35,884 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 03:37:41,099 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 36.17 2021-05-03 03:37:41,371 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:aiFu8VpTTc%-22-658000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-477 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:41,371 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in bucket1:Y8W304c5dMMguaKw-22-523000:aiFu8VpTTc%-22-658000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-477 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:43,102 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:43,104 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1366 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:43,141 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:43,141 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1366 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:46,266 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 39.73 2021-05-03 03:37:51,538 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 43.86 2021-05-03 03:37:55,332 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,334 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,336 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,338 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,427 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,428 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,436 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,437 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1630 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,437 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1630 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:55,437 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1739 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:37:56,767 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 47.61 2021-05-03 03:38:01,980 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 51.03 2021-05-03 03:38:07,359 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 54.19 2021-05-03 03:38:12,605 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 58.12 2021-05-03 03:38:14,578 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,581 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,582 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,584 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,596 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,598 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,598 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,598 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,605 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:14,607 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:17,857 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 61.61 2021-05-03 03:38:19,006 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,007 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,012 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,013 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,013 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,013 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,016 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,016 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,016 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:19,017 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2429 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:23,028 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 64.78 2021-05-03 03:38:25,851 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,851 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,854 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2624 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2680 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,852 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2624 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,858 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,859 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2624 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2680 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,867 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,868 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2624 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2680 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,878 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:25,878 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2710 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2624 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2680 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:30,147 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 03:38:34,404 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:XGrohj82-18-421000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:35,302 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 71.44 2021-05-03 03:38:40,783 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 74.65 2021-05-03 03:38:40,914 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:K6OW1kBigy6L%-EOaDw-18-419000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-228 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-368 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-377 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-221 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-210 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-332 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-380 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:41,923 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:Mj64hE-18-406000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-228 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-368 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-377 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-221 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-210 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-332 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-380 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:45,911 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 78.37 2021-05-03 03:38:46,306 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:Mj64hE-18-406000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-859 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-850 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:50,664 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:XGrohj82-18-421000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-418 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-408 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-405 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-403 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-448 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-489 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-454 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-410 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-411 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-486 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-464 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-472 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-440 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:50,900 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:XGrohj82-18-421000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-909 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-958 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-972 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:51,144 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 82.26 2021-05-03 03:38:51,910 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:1HYI-HzwtCYz7AwJPIZj-18-417000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-309 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-326 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-359 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-316 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-368 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-377 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-334 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-221 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-210 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-299 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-332 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-286 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-380 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:51,910 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:T-18-415000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-309 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-326 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-359 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-316 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-368 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-377 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-334 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-210 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-299 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-332 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-286 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-380 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:52,329 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:T-18-415000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-801 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-856 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-813 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-844 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-734 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-899 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-742 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-874 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-850 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-793 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-782 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:54,145 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:K6OW1kBigy6L%-EOaDw-18-419000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-859 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-850 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:56,125 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:R6aBIcoJm2SK-18-407000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-845 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-856 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-813 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-844 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-734 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-899 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-742 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-874 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-831 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-850 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-782 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:56,765 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:nk8bz-18-418000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-519 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-636 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-526 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-598 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-664 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-562 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-661 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-673 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:38:56,819 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 85.61 2021-05-03 03:39:01,181 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:w2mZgrCb-24-230000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1378 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:01,183 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:w2mZgrCb-24-230000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1378 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:02,098 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 89.13 2021-05-03 03:39:02,575 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:Mj64hE-18-406000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-927 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-903 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:02,944 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:IxKZVvGupbhyo-18-411000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-519 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-647 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-513 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-522 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-654 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-599 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-676 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-556 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-611 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-534 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-633 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-642 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-571 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:04,286 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:IxKZVvGupbhyo-18-411000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-69 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-44 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-118 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-104 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-81 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-112 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-175 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-198 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-163 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-141 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:04,631 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:K6OW1kBigy6L%-EOaDw-18-419000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-409 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-403 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-489 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-424 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-410 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-464 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-483 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-462 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:04,828 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:Mj64hE-18-406000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-409 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-427 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-403 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-489 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-410 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-464 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-483 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-462 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:06,038 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:BdlUJ9gF5zWO3tjfV-18-410000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:06,039 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:BdlUJ9gF5zWO3tjfV-18-410000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2869 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:07,375 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 91.97 2021-05-03 03:39:08,749 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:aijOl%U-bf-24-215000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1224 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1208 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1364 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1398 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:08,750 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:aijOl%U-bf-24-215000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1224 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1208 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1364 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1398 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:09,322 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:T-18-415000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-427 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-449 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-437 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-415 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-424 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-450 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-461 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-462 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:09,323 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:1HYI-HzwtCYz7AwJPIZj-18-417000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-427 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-449 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-437 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-415 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-424 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-450 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-461 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-462 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:09,680 | test | INFO | pool-2-thread-23 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:T-18-415000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:10,875 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:R6aBIcoJm2SK-18-407000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:10,877 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:K6OW1kBigy6L%-EOaDw-18-419000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-922 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:11,059 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:nk8bz-18-418000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-792 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:12,644 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 95.43 2021-05-03 03:39:13,046 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:w2mZgrCb-24-230000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1543 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1587 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1417 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:13,049 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:w2mZgrCb-24-230000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1554 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1587 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1543 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1501 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1417 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1491 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:13,213 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:IxKZVvGupbhyo-18-411000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-835 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-744 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-861 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:14,658 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-03 03:39:14,661 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 10 docs from 0 to 5000 2021-05-03 03:39:14,664 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 10 docs from 0 to 5000 2021-05-03 03:39:14,664 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 10 docs from 0 to 5000 2021-05-03 03:39:14,665 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-03 03:39:14,667 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-03 03:39:14,687 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:IxKZVvGupbhyo-18-411000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-219 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-337 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-312 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-321 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-282 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:16,266 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:tPQKj2_BT6-18-421000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-623 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-620 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:16,732 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:scope1:0GvCE-18-390000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-542 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-543 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-681 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:17,776 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 98.35 2021-05-03 03:39:17,937 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:0GvCE-18-390000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-36 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-14 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-51 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-148 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-60 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:18,767 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:eFwbcgtUX6-18-420000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-36 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-14 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-77 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-66 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-57 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-51 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-148 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:19,884 | test | INFO | pool-2-thread-14 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:dd6WL9NnFW-18-417000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-696 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-531 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-584 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:20,499 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:aijOl%U-bf-24-215000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1406 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1596 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:20,500 | test | INFO | pool-2-thread-27 | [table_view:display:72] Keys failed in default:zMgQtox-24-52000:aijOl%U-bf-24-215000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1406 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1596 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:20,934 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:L9A0yeM2OnZbBL9K36-18-415000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-512 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-531 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:21,101 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:ZU7qUrZ788YuT-18-408000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-638 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-531 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:21,112 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:CP4ub4RVMxVFFp-23-646000:rEFIL2AbvIU5%j4-23-765000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:21,115 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:CP4ub4RVMxVFFp-23-646000:rEFIL2AbvIU5%j4-23-765000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-94 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:21,904 | test | INFO | pool-2-thread-16 | [table_view:display:72] Keys failed in default:scope1:PQ2hVLL-NtGmbkTQY-18-392000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-29 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-66 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-57 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-147 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-194 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-181 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:22,042 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:scope1:PQ2hVLL-NtGmbkTQY-18-392000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-557 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-635 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-531 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-560 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:22,049 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:scope1:PQ2hVLL-NtGmbkTQY-18-392000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2546 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:22,052 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:scope1:PQ2hVLL-NtGmbkTQY-18-392000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2546 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:22,325 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:IxKZVvGupbhyo-18-411000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-933 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-976 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:23,986 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in default:scope1:fspk-iFgX5fC7C56-24-800000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:23,987 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in default:scope1:fspk-iFgX5fC7C56-24-800000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-137 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:24,332 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:scope1:0GvCE-18-390000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-710 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:26,003 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:bq42%COmiN8CQ58b-18-406000:nk8bz-18-418000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-931 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 03:39:27,973 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 03:39:28,035 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 170.761000156 sec 2021-05-03 03:43:18,910 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 03:43:18,911 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-03 03:43:18,914 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 03:43:18,917 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 4 docs from 500 to 980 2021-05-03 03:43:18,917 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 03:43:18,918 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 16 docs from 0 to 500 2021-05-03 03:43:18,920 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 21 docs from 0 to 500 2021-05-03 03:43:18,921 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 03:43:18,921 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 21 docs from 0 to 500 2021-05-03 03:43:18,923 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 30 docs from 0 to 500 2021-05-03 03:43:18,924 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 21 docs from 500 to 980 2021-05-03 03:43:18,926 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 29 docs from 0 to 500 2021-05-03 03:43:18,927 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 20 docs from 500 to 980 2021-05-03 03:43:18,927 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 03:43:18,928 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 28 docs from 500 to 980 2021-05-03 03:43:18,930 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 33 docs from 500 to 980 2021-05-03 03:43:18,931 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 28 docs from 0 to 500 2021-05-03 03:43:18,933 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-03 03:43:18,933 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-03 03:43:18,934 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 03:43:18,936 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 6 docs from 500 to 980 2021-05-03 03:43:53,098 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 03:43:53,099 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 3 docs from 500 to 980 2021-05-03 03:43:53,101 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 03:43:53,101 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 10 docs from 0 to 500 2021-05-03 03:43:53,102 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-03 03:43:53,104 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-03 03:43:53,105 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 03:43:53,105 | test | ERROR | pool-2-thread-18 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 03:47:08,858 | test | INFO | pool-2-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****...*.....*...X | 13.800 | | 12.600 | .........X | 12.600 | | 5.900 | ...X | 5.900 | | 2.700 | .X | 2.700 | | 0.000 | **....*........*.....X | 21.106 | | 20.599 | ...............X | 20.599 | | 19.999 | ...............**.........*...*.................X | 56.581 | | 51.787 | ........................................*....X | 55.885 | | 47.815 | .....................................*X | 47.815 | | 44.212 | ..................................X | 44.212 | | 39.812 | ...............................*...X | 43.500 | | 42.794 | .................................*..*.................**......*.....X | 79.174 | | 77.474 | .............................................................*.**.........*.......**.X | 100.092 | | 96.092 | ............................................................................X | 96.092 | | 84.289 | ...................................................................*X | 84.289 | | 81.703 | .................................................................X | 81.703 | | 75.196 | ...........................................................*....X | 80.005 | | 79.798 | ...............................................................X | 79.798 | | 76.893 | .............................................................X | 76.893 | | 76.188 | ............................................................*X | 76.188 | | 74.788 | ...........................................................*...*X | 78.615 | | 74.412 | ...........................................................X | 74.412 | | 55.496 | ...........................................*X | 55.496 | | 55.003 | ...........................................**.....................**......X | 88.900 | | 88.300 | ......................................................................X | 88.300 | | 82.900 | ..................................................................*X | 82.900 | | 75.900 | ............................................................*X | 75.900 | | 36.197 | ............................*X | 36.197 | | 24.197 | ..................*X | 24.197 | | 15.497 | ...........X | 15.497 | | 0.200 | *X | 0.200 | | 0.000 | ***************.**X | 1.900 | | 1.799 | **....*.................**..........**..............................***...***..**X | 83.985 | | 82.685 | .................................................................X | 82.685 | | 75.604 | ............................................................X | 75.604 | | 75.514 | ............................................................****.....**X | 82.700 | | 77.700 | .............................................................X | 77.700 | | 60.400 | ...............................................*X | 60.400 | | 55.094 | ...........................................*.....**....................*...**.X | 91.199 | | 82.499 | .................................................................*X | 82.499 | | 79.191 | ...............................................................*X | 79.191 | | 54.004 | ..........................................*X | 54.004 | | 52.204 | .........................................***........**.*X | 63.903 | | 63.295 | ..................................................**...*X | 67.295 | | 65.687 | ....................................................*..*X | 69.199 | | 65.590 | ....................................................*...*.....**...*X | 79.413 | | 79.203 | ...............................................................*........*X | 89.921 | | 89.905 | .......................................................................*X | 89.905 | | 39.588 | ...............................*X | 39.588 | | 24.393 | ..................*X | 24.393 | | 16.098 | ............X | 16.098 | | 2.700 | .*X | 2.700 | | 2.500 | .**..*...**X | 9.704 | | 8.104 | .....*X | 8.104 | | 6.500 | ....**X | 6.600 | | 6.000 | ...X | 6.000 | | 3.100 | .*X | 3.100 | | 2.899 | .**..*X | 5.398 | | 4.699 | ..*X | 4.699 | | 2.798 | .X | 2.798 | | 0.400 | *......**.............*................................................**.........**.........*...........*....*X | 123.588 | | 117.988 | ..............................................................................................X | 117.988 | | 107.413 | .....................................................................................*X | 107.413 | | 104.413 | ...................................................................................*X | 104.413 | | 96.580 | .............................................................................*X | 96.580 | | 91.480 | .........................................................................*X | 91.480 | | 88.180 | ......................................................................*.......*X | 96.423 | | 93.223 | ..........................................................................X | 93.223 | | 79.523 | ...............................................................X | 79.523 | | 22.200 | ................*X | 22.200 | | 10.399 | .......**.......*..................................*.................**...*............*X | 101.122 | | 98.806 | ..............................................................................**....X | 103.706 | | 102.800 | ..................................................................................**X | 103.400 | | 100.794 | ................................................................................*.....***X | 107.494 | | 105.796 | ....................................................................................X | 105.796 | | 93.482 | ..........................................................................X | 93.482 | | 91.882 | .........................................................................*X | 91.882 | | 83.091 | ..................................................................X | 83.091 | | 54.512 | ...........................................*X | 54.512 | | 36.011 | ............................*X | 36.011 | | 29.206 | ......................X | 29.206 | | 10.098 | .......**X | 10.601 | | 8.901 | ......X | 8.901 | | 7.602 | .....**..**.X | 11.599 | | 10.199 | .......X | 10.199 | | 5.800 | ...*X | 5.800 | | 4.300 | ..X | 4.300 | | 3.500 | .**...**....*X | 11.500 | | 9.100 | ......*X | 9.100 | | 7.000 | ....X | 7.000 | | 3.900 | ..**.*..**X | 7.899 | | 5.699 | ...X | 5.699 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 03:47:09,944 | test | INFO | pool-2-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....*....*..X | 32.700 | | 26.800 | .......X | 26.800 | | 26.784 | .......X | 26.784 | | 24.484 | ......*...*X | 34.184 | | 18.801 | .....*.....**.............*...**...............*..........*.........*.*X | 195.736 | | 157.521 | .................................................X | 157.521 | | 153.098 | ...............................................X | 153.098 | | 118.085 | ....................................*...........X | 152.585 | | 139.000 | ...........................................**..........X | 170.855 | | 135.155 | ..........................................*.............X | 175.455 | | 142.822 | ............................................**......X | 160.714 | | 146.714 | .............................................*.........X | 174.992 | | 144.341 | .............................................*..........X | 175.341 | | 142.941 | ............................................**.............**.........X | 212.666 | | 208.199 | .................................................................*..**..*......*........................X | 312.967 | | 306.648 | ................................................................................................**.X | 309.148 | | 297.059 | .............................................................................................X | 297.059 | | 295.542 | .............................................................................................*.*.*X | 301.483 | | 283.512 | .........................................................................................X | 283.512 | | 70.512 | .....................*X | 70.512 | | 31.000 | ........X | 31.000 | | 13.000 | ...X | 13.000 | | 0.000 | ****...............**......**..**.......**X | 95.912 | | 90.112 | ...........................*..**..**...X | 109.692 | | 107.908 | .................................X | 107.908 | | 100.299 | ...............................*X | 100.299 | | 92.599 | ............................*.**.*X | 98.401 | | 78.007 | .......................*X | 78.007 | | 61.796 | ..................*X | 61.796 | | 44.507 | .............*X | 44.507 | | 4.600 | X | 4.600 | | 0.300 | ******X | 2.001 | | 0.200 | **X | 0.200 | | 0.000 | *****.**....*...**..*........**..**.**X | 68.192 | | 61.004 | ..................X | 61.004 | | 58.599 | .................X | 58.599 | | 57.299 | .................**.*X | 60.194 | | 59.904 | ..................***X | 60.104 | | 59.998 | ..................*.**..**..*X | 76.292 | | 74.392 | ......................*X | 74.392 | | 70.485 | .....................X | 70.485 | | 63.799 | ...................*X | 63.799 | | 56.693 | .................*X | 56.693 | | 53.205 | ................X | 53.205 | | 40.195 | ...........*X | 40.195 | | 36.604 | ..........*X | 36.604 | | 4.703 | *X | 4.703 | | 0.400 | *******.*X | 5.600 | | 5.000 | *X | 5.000 | | 3.599 | X | 3.599 | | 2.501 | ***X | 2.701 | | 1.701 | ****.X | 4.100 | | 2.700 | ****.X | 5.998 | | 5.998 | *X | 5.998 | | 5.699 | X | 5.699 | | 4.298 | X | 4.298 | | 2.701 | ****.*X | 3.900 | | 3.100 | *X | 3.100 | | 2.100 | X | 2.100 | | 0.500 | X | 0.500 | | 0.200 | *X | 0.200 | | 0.000 | ******.**X | 5.099 | | 3.699 | *X | 3.699 | | 2.200 | X | 2.200 | | 1.600 | *X | 1.600 | | 1.500 | ***.*****X | 5.801 | | 5.601 | X | 5.601 | | 4.101 | *X | 4.101 | | 2.100 | *X | 2.100 | | 0.800 | ********X | 1.900 | | 1.700 | X | 1.700 | | 1.700 | *****X | 3.101 | | 2.000 | *X | 2.000 | | 1.300 | X | 1.300 | | 1.100 | ***.***.*X | 8.800 | | 7.100 | .X | 7.100 | | 4.799 | *X | 4.799 | | 1.400 | X | 1.400 | | 0.700 | *******X | 3.100 | | 1.800 | X | 1.800 | | 1.300 | X | 1.300 | | 0.600 | *****X | 1.000 | | 0.800 | *X | 0.800 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 03:47:11,992 | test | INFO | pool-2-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.*.**..X | 52.500 | | 36.700 | ..X | 36.700 | | 35.700 | ..*X | 35.700 | | 32.200 | .*X | 32.200 | | 25.400 | .*.X | 45.320 | | 36.220 | ..*X | 36.220 | | 34.720 | ..*.......*...........*......................**....X | 547.282 | | 526.406 | ............................................X | 526.406 | | 320.666 | ..........................X | 320.666 | | 217.733 | ..................X | 217.733 | | 171.111 | .............*...**..*.X | 231.400 | | 220.854 | ..................X | 220.854 | | 163.054 | .............X | 163.054 | | 145.954 | ...........*X | 145.954 | | 138.441 | ...........*..X | 166.920 | | 159.920 | ............X | 159.920 | | 128.579 | ..........*.*.....*.........**.*X | 314.261 | | 272.035 | ......................X | 272.035 | | 256.107 | .....................X | 256.107 | | 210.307 | .................X | 210.307 | | 192.153 | ...............*X | 192.153 | | 170.020 | .............*....X | 209.150 | | 168.130 | .............X | 168.130 | | 141.130 | ...........X | 141.130 | | 95.228 | .......X | 95.228 | | 90.521 | ......*X | 90.521 | | 81.521 | ......*.X | 92.893 | | 90.206 | ......**..............**............*......*.............*X | 604.600 | | 561.000 | ...............................................**....................**...X | 819.292 | | 819.292 | ......................................................................*..X | 839.092 | | 675.675 | .........................................................X | 675.675 | | 652.175 | .......................................................*X | 652.175 | | 650.975 | .......................................................*.........**....*X | 793.538 | | 744.038 | ...............................................................X | 744.038 | | 668.448 | .........................................................X | 668.448 | | 620.787 | .....................................................*X | 620.787 | | 603.187 | ...................................................*X | 603.187 | | 510.591 | ...........................................*X | 510.591 | | 507.311 | ...........................................*.....*.................**....*X | 806.993 | | 677.083 | ..........................................................*X | 677.083 | | 601.883 | ...................................................*X | 601.883 | | 521.893 | ............................................X | 521.893 | | 343.454 | ............................*X | 343.454 | | 336.554 | ............................**..*.......*.*X | 450.721 | | 412.321 | ..................................*X | 412.321 | | 361.320 | ..............................X | 361.320 | | 355.174 | ..............................**...*.....*X | 453.326 | | 441.200 | .....................................*X | 441.200 | | 384.300 | ................................**.**..*...*.......**X | 537.844 | | 483.713 | .........................................X | 483.713 | | 330.213 | ...........................*..**.*...*X | 397.819 | | 357.819 | ..............................*X | 357.819 | | 352.982 | .............................*.....X | 409.916 | | 404.216 | ..................................**.X | 412.916 | | 276.975 | .......................***X | 284.475 | | 256.546 | .....................*X | 261.896 | | 240.266 | ...................*X | 240.266 | | 236.023 | ...................**...*..**..*..*.**......**........................*..........X | 802.113 | | 802.113 | .....................................................................*....X | 854.772 | | 849.389 | .........................................................................X | 849.389 | | 828.631 | .......................................................................**...*..................**.......*X | 1145.413 | | 1115.813 | ................................................................................................X | 1115.813 | | 979.000 | ....................................................................................X | 979.000 | | 880.000 | ...........................................................................*X | 880.000 | | 852.652 | .........................................................................X | 852.652 | | 710.852 | .............................................................*X | 710.852 | | 698.884 | ............................................................*X | 698.884 | | 602.974 | ...................................................*X | 602.974 | | 596.907 | ...................................................X | 596.907 | | 596.907 | ...................................................**.......*.*X | 691.422 | | 664.422 | .........................................................X | 664.422 | | 532.600 | .............................................*X | 532.600 | | 439.600 | .....................................*X | 439.600 | | 392.551 | .................................**......***....X | 505.759 | | 505.759 | ...........................................X | 505.759 | | 470.459 | ........................................*X | 470.459 | | 422.005 | ...................................*X | 422.005 | | 410.805 | ..................................*X | 410.805 | | 386.205 | ................................X | 386.205 | | 365.681 | ..............................**.*.....*............X | 565.800 | | 565.800 | ................................................*........*X | 661.178 | | 652.878 | .......................................................*...X | 679.538 | | 604.029 | ...................................................*X | 604.029 | | 577.829 | .................................................*X | 577.829 | | 428.690 | ....................................X | 428.690 | | 399.321 | .................................*X | 399.321 | | 386.321 | ................................X | 386.321 | | 372.600 | ...............................**.*X | 379.000 | | 358.282 | ..............................X | 358.282 | | 333.382 | ............................X | 333.382 | | 319.082 | ..........................**.X | 324.987 | | 319.834 | ..........................**.*X | 321.434 | | 318.578 | ..........................X | 318.578 | | 317.789 | ..........................**....*....*..............**..**..X | 607.861 | | 587.709 | ..................................................*X | 587.709 | | 581.899 | .................................................X | 581.899 | | 580.399 | .................................................*........**...**..X | 723.131 | | 714.131 | .............................................................*X | 714.131 | | 684.400 | ..........................................................*.*...**.....**...*....**...*..*.............*X | 1069.601 | | 1044.182 | ..........................................................................................*X | 1044.182 | +----------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 03:47:12,161 | test | WARNING | Thread-2419 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:Y8W304c5dMMguaKw-22-523000:HhUXyBP56XIwcU8sHiBW-22-657000 create failed keys from task: [u'test_collections-704', u'test_collections-302', u'test_collections-676', u'test_collections-215', u'test_collections-694', u'test_collections-365', u'test_collections-750', u'test_collections-385', u'test_collections-292'] 2021-05-03 03:47:12,163 | test | WARNING | Thread-2420 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:Y8W304c5dMMguaKw-22-523000:aiFu8VpTTc%-22-658000 create failed keys from task: [u'test_collections-303', u'test_collections-477', u'test_collections-111', u'test_collections-133', u'test_collections-82', u'test_collections-293'] 2021-05-03 03:47:13,158 | test | WARNING | Thread-2650 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:V3XOP_bFNBCqWS3%-21-786000:8-21-938000 create failed keys from task: [u'test_collections-2391', u'test_collections-1739', u'test_collections-2710', u'test_collections-2624', u'test_collections-2429', u'test_collections-2680', u'test_collections-2661', u'test_collections-1663', u'test_collections-1630', u'test_collections-2201'] 2021-05-03 03:47:13,158 | test | WARNING | Thread-2651 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:V3XOP_bFNBCqWS3%-21-786000:kogRWZ86pORMO-24-801000 create failed keys from task: [u'test_collections-2391', u'test_collections-1739', u'test_collections-2710', u'test_collections-2624', u'test_collections-2429', u'test_collections-2680', u'test_collections-2661', u'test_collections-1366', u'test_collections-1663', u'test_collections-2201'] 2021-05-03 03:47:13,164 | test | WARNING | Thread-2652 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:V3XOP_bFNBCqWS3%-21-786000:nkuJmVVr-21-943000 create failed keys from task: [u'test_collections-2391', u'test_collections-1739', u'test_collections-2710', u'test_collections-2624', u'test_collections-2429', u'test_collections-2680', u'test_collections-2661', u'test_collections-1366', u'test_collections-1663', u'test_collections-2201'] 2021-05-03 03:47:13,253 | test | WARNING | Thread-2728 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:eaIBnjdGLi8yG8e-22-158000:byzM99OpZlelzsSko-22-302000 create failed keys from task: [u'test_collections-2391', u'test_collections-1739', u'test_collections-2624', u'test_collections-2429', u'test_collections-2661', u'test_collections-1663', u'test_collections-2201'] 2021-05-03 03:47:13,255 | test | WARNING | Thread-2729 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:eaIBnjdGLi8yG8e-22-158000:xB5gg4TqeNnoHTHVpS-S-22-303000 create failed keys from task: [u'test_collections-2391', u'test_collections-1739', u'test_collections-2710', u'test_collections-2624', u'test_collections-2429', u'test_collections-2680', u'test_collections-2661', u'test_collections-1663', u'test_collections-2201'] 2021-05-03 03:47:13,257 | test | WARNING | Thread-2733 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:Mj64hE-18-406000 update failed keys from task: [u'test_collections-729', u'test_collections-927', u'test_collections-859', u'test_collections-815', u'test_collections-903', u'test_collections-769', u'test_collections-832', u'test_collections-896', u'test_collections-850', u'test_collections-880'] 2021-05-03 03:47:13,259 | test | WARNING | Thread-2735 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:T-18-415000 update failed keys from task: [u'test_collections-808', u'test_collections-729', u'test_collections-801', u'test_collections-769', u'test_collections-844', u'test_collections-742', u'test_collections-862', u'test_collections-782', u'test_collections-880', u'test_collections-815', u'test_collections-856', u'test_collections-813', u'test_collections-978', u'test_collections-734', u'test_collections-899', u'test_collections-896', u'test_collections-797', u'test_collections-874', u'test_collections-850', u'test_collections-793', u'test_collections-770'] 2021-05-03 03:47:13,259 | test | WARNING | Thread-2736 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:BdlUJ9gF5zWO3tjfV-18-410000 create failed keys from task: [u'test_collections-2802', u'test_collections-2869'] 2021-05-03 03:47:13,260 | test | WARNING | Thread-2738 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:eFwbcgtUX6-18-420000 delete failed keys from task: [u'test_collections-36', u'test_collections-14', u'test_collections-77', u'test_collections-66', u'test_collections-57', u'test_collections-73', u'test_collections-51', u'test_collections-148', u'test_collections-187'] 2021-05-03 03:47:13,262 | test | WARNING | Thread-2740 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:K6OW1kBigy6L%-EOaDw-18-419000 update failed keys from task: [u'test_collections-729', u'test_collections-859', u'test_collections-815', u'test_collections-922', u'test_collections-769', u'test_collections-978', u'test_collections-832', u'test_collections-896', u'test_collections-850', u'test_collections-880'] 2021-05-03 03:47:13,263 | test | WARNING | Thread-2741 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:XGrohj82-18-421000 update failed keys from task: [u'test_collections-909', u'test_collections-958', u'test_collections-832', u'test_collections-972'] 2021-05-03 03:47:13,265 | test | WARNING | Thread-2743 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:IxKZVvGupbhyo-18-411000 update failed keys from task: [u'test_collections-607', u'test_collections-647', u'test_collections-527', u'test_collections-648', u'test_collections-843', u'test_collections-602', u'test_collections-921', u'test_collections-522', u'test_collections-720', u'test_collections-688', u'test_collections-642', u'test_collections-760', u'test_collections-933', u'test_collections-976', u'test_collections-654', u'test_collections-611', u'test_collections-534', u'test_collections-571', u'test_collections-702', u'test_collections-824', u'test_collections-744', u'test_collections-861', u'test_collections-660', u'test_collections-580', u'test_collections-519', u'test_collections-835', u'test_collections-513', u'test_collections-679', u'test_collections-599', u'test_collections-676', u'test_collections-556', u'test_collections-633', u'test_collections-593'] 2021-05-03 03:47:13,266 | test | WARNING | Thread-2745 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:R6aBIcoJm2SK-18-407000 update failed keys from task: [u'test_collections-808', u'test_collections-729', u'test_collections-845', u'test_collections-769', u'test_collections-844', u'test_collections-742', u'test_collections-862', u'test_collections-782', u'test_collections-815', u'test_collections-856', u'test_collections-813', u'test_collections-978', u'test_collections-734', u'test_collections-899', u'test_collections-896', u'test_collections-797', u'test_collections-874', u'test_collections-831', u'test_collections-850', u'test_collections-770'] 2021-05-03 03:47:13,266 | test | WARNING | Thread-2746 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:1HYI-HzwtCYz7AwJPIZj-18-417000 delete failed keys from task: [u'test_collections-449', u'test_collections-326', u'test_collections-368', u'test_collections-362', u'test_collections-286', u'test_collections-415', u'test_collections-259', u'test_collections-377', u'test_collections-334', u'test_collections-210', u'test_collections-299', u'test_collections-332', u'test_collections-450', u'test_collections-371', u'test_collections-290', u'test_collections-309', u'test_collections-427', u'test_collections-424', u'test_collections-300', u'test_collections-221', u'test_collections-461', u'test_collections-264', u'test_collections-462', u'test_collections-380', u'test_collections-359', u'test_collections-316', u'test_collections-437', u'test_collections-232', u'test_collections-474', u'test_collections-431'] 2021-05-03 03:47:13,289 | test | WARNING | Thread-2753 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:nk8bz-18-418000 update failed keys from task: [u'test_collections-607', u'test_collections-805', u'test_collections-926', u'test_collections-526', u'test_collections-648', u'test_collections-562', u'test_collections-561', u'test_collections-938', u'test_collections-931', u'test_collections-698', u'test_collections-821', u'test_collections-621', u'test_collections-864', u'test_collections-664', u'test_collections-661', u'test_collections-580', u'test_collections-719', u'test_collections-519', u'test_collections-636', u'test_collections-834', u'test_collections-679', u'test_collections-798', u'test_collections-872', u'test_collections-598', u'test_collections-673', u'test_collections-871', u'test_collections-593', u'test_collections-792'] 2021-05-03 03:47:13,342 | test | WARNING | Thread-2754 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:ZU7qUrZ788YuT-18-408000 update failed keys from task: [u'test_collections-638', u'test_collections-549', u'test_collections-531'] 2021-05-03 03:47:13,438 | test | WARNING | Thread-2735 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:T-18-415000 delete failed keys from task: [u'test_collections-449', u'test_collections-326', u'test_collections-368', u'test_collections-362', u'test_collections-286', u'test_collections-415', u'test_collections-259', u'test_collections-377', u'test_collections-334', u'test_collections-210', u'test_collections-299', u'test_collections-332', u'test_collections-450', u'test_collections-371', u'test_collections-290', u'test_collections-309', u'test_collections-427', u'test_collections-424', u'test_collections-300', u'test_collections-461', u'test_collections-264', u'test_collections-462', u'test_collections-380', u'test_collections-359', u'test_collections-316', u'test_collections-437', u'test_collections-232', u'test_collections-474', u'test_collections-431'] 2021-05-03 03:47:13,588 | test | WARNING | Thread-2740 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:K6OW1kBigy6L%-EOaDw-18-419000 delete failed keys from task: [u'test_collections-409', u'test_collections-403', u'test_collections-228', u'test_collections-368', u'test_collections-489', u'test_collections-446', u'test_collections-424', u'test_collections-221', u'test_collections-464', u'test_collections-483', u'test_collections-264', u'test_collections-462', u'test_collections-380', u'test_collections-478', u'test_collections-377', u'test_collections-410', u'test_collections-455', u'test_collections-232', u'test_collections-210', u'test_collections-474', u'test_collections-332'] 2021-05-03 03:47:13,657 | test | WARNING | Thread-2733 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:Mj64hE-18-406000 delete failed keys from task: [u'test_collections-409', u'test_collections-427', u'test_collections-403', u'test_collections-228', u'test_collections-368', u'test_collections-489', u'test_collections-446', u'test_collections-221', u'test_collections-464', u'test_collections-483', u'test_collections-264', u'test_collections-462', u'test_collections-380', u'test_collections-478', u'test_collections-377', u'test_collections-410', u'test_collections-455', u'test_collections-232', u'test_collections-210', u'test_collections-474', u'test_collections-332'] 2021-05-03 03:47:13,703 | test | WARNING | Thread-2743 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:IxKZVvGupbhyo-18-411000 delete failed keys from task: [u'test_collections-98', u'test_collections-55', u'test_collections-126', u'test_collections-97', u'test_collections-321', u'test_collections-163', u'test_collections-282', u'test_collections-44', u'test_collections-219', u'test_collections-337', u'test_collections-135', u'test_collections-175', u'test_collections-130', u'test_collections-170', u'test_collections-104', u'test_collections-346', u'test_collections-389', u'test_collections-81', u'test_collections-141', u'test_collections-2', u'test_collections-69', u'test_collections-118', u'test_collections-159', u'test_collections-312', u'test_collections-112', u'test_collections-198', u'test_collections-275', u'test_collections-192'] 2021-05-03 03:47:13,713 | test | WARNING | Thread-2741 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:XGrohj82-18-421000 delete failed keys from task: [u'test_collections-408', u'test_collections-405', u'test_collections-403', u'test_collections-448', u'test_collections-489', u'test_collections-446', u'test_collections-486', u'test_collections-464', u'test_collections-440', u'test_collections-418', u'test_collections-438', u'test_collections-478', u'test_collections-454', u'test_collections-410', u'test_collections-411', u'test_collections-472'] 2021-05-03 03:47:15,322 | test | WARNING | Thread-2849 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:0GvCE-18-390000 update failed keys from task: [u'test_collections-726', u'test_collections-549', u'test_collections-542', u'test_collections-543', u'test_collections-763', u'test_collections-681', u'test_collections-618', u'test_collections-578', u'test_collections-710', u'test_collections-631', u'test_collections-692'] 2021-05-03 03:47:15,325 | test | WARNING | Thread-2852 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:PQ2hVLL-NtGmbkTQY-18-392000 update failed keys from task: [u'test_collections-528', u'test_collections-557', u'test_collections-635', u'test_collections-586', u'test_collections-531', u'test_collections-574', u'test_collections-560'] 2021-05-03 03:47:15,334 | test | WARNING | Thread-2871 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:fspk-iFgX5fC7C56-24-800000 create failed keys from task: [u'test_collections-137'] 2021-05-03 03:47:15,349 | test | WARNING | Thread-2876 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:CP4ub4RVMxVFFp-23-646000:rEFIL2AbvIU5%j4-23-765000 create failed keys from task: [u'test_collections-14', u'test_collections-94'] 2021-05-03 03:47:15,365 | test | WARNING | Thread-2760 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:dd6WL9NnFW-18-417000 update failed keys from task: [u'test_collections-549', u'test_collections-696', u'test_collections-531', u'test_collections-584', u'test_collections-692'] 2021-05-03 03:47:15,382 | test | WARNING | Thread-2801 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zMgQtox-24-52000:w2mZgrCb-24-230000 create failed keys from task: [u'test_collections-1378', u'test_collections-1554', u'test_collections-1587', u'test_collections-1543', u'test_collections-1501', u'test_collections-1417', u'test_collections-1491'] 2021-05-03 03:47:15,387 | test | WARNING | Thread-2852 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:PQ2hVLL-NtGmbkTQY-18-392000 delete failed keys from task: [u'test_collections-29', u'test_collections-66', u'test_collections-12', u'test_collections-57', u'test_collections-147', u'test_collections-194', u'test_collections-181'] 2021-05-03 03:47:15,388 | test | WARNING | Thread-2761 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:tPQKj2_BT6-18-421000 update failed keys from task: [u'test_collections-618', u'test_collections-549', u'test_collections-623', u'test_collections-578', u'test_collections-620', u'test_collections-692'] 2021-05-03 03:47:15,397 | test | WARNING | Thread-2765 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bq42%COmiN8CQ58b-18-406000:L9A0yeM2OnZbBL9K36-18-415000 update failed keys from task: [u'test_collections-549', u'test_collections-512', u'test_collections-531'] 2021-05-03 03:47:15,403 | test | WARNING | Thread-2800 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zMgQtox-24-52000:aijOl%U-bf-24-215000 create failed keys from task: [u'test_collections-1224', u'test_collections-1208', u'test_collections-1406', u'test_collections-1364', u'test_collections-1398', u'test_collections-1596'] 2021-05-03 03:47:15,418 | test | WARNING | Thread-2852 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:PQ2hVLL-NtGmbkTQY-18-392000 create failed keys from task: [u'test_collections-2546'] 2021-05-03 03:47:15,453 | test | WARNING | Thread-2849 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:0GvCE-18-390000 delete failed keys from task: [u'test_collections-36', u'test_collections-14', u'test_collections-95', u'test_collections-73', u'test_collections-51', u'test_collections-148', u'test_collections-168', u'test_collections-60', u'test_collections-187', u'test_collections-9'] 2021-05-03 03:47:31,311 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 03:49:20,397 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 03:49:20,400 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 03:49:41,743 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 03:49:41,744 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-03 03:50:12,884 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 03:50:12,885 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 03:51:06,884 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 03:51:24,273 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2170000076 seconds to complete 2021-05-03 03:51:24,345 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2664.016s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_1 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_2', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 2, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 03:51:34,710 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 03:51:35,747 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 03:51:36,308 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde2 type=client open_count=1 channel=[id: 0x45c5d461, 0.0.0.0/0.0.0.0:39245] timeout=300.0>) 2021-05-03 03:51:42,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde3 type=client open_count=1 channel=[id: 0xbd1b9aa1, 0.0.0.0/0.0.0.0:59295] timeout=300.0>) 2021-05-03 03:51:44,161 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde4 type=client open_count=1 channel=[id: 0x181e6dd8, 0.0.0.0/0.0.0.0:38073] timeout=300.0>) 2021-05-03 03:51:46,184 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde5 type=client open_count=1 channel=[id: 0xfd078f9e, 0.0.0.0/0.0.0.0:50513] timeout=300.0>) 2021-05-03 03:51:48,210 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde6 type=client open_count=1 channel=[id: 0x6fcb2704, 0.0.0.0/0.0.0.0:58629] timeout=300.0>) 2021-05-03 03:51:50,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde7 type=client open_count=1 channel=[id: 0xcdc9ee79, 0.0.0.0/0.0.0.0:55349] timeout=300.0>) 2021-05-03 03:51:52,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde8 type=client open_count=1 channel=[id: 0xe4f80cad, 0.0.0.0/0.0.0.0:54963] timeout=300.0>) 2021-05-03 03:51:54,290 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cde9 type=client open_count=1 channel=[id: 0xfc21417b, 0.0.0.0/0.0.0.0:55089] timeout=300.0>) 2021-05-03 03:51:56,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5cdea type=client open_count=1 channel=[id: 0x76dc3def, 0.0.0.0/0.0.0.0:48213] timeout=300.0>) 2021-05-03 03:51:58,351 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdeb type=client open_count=1 channel=[id: 0x63235f43, 0.0.0.0/0.0.0.0:60241] timeout=300.0>) 2021-05-03 03:52:05,750 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdec type=client open_count=1 channel=[id: 0xb5c2a360, 0.0.0.0/0.0.0.0:56225] timeout=300.0>) 2021-05-03 03:52:07,773 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cded type=client open_count=1 channel=[id: 0x4d126603, 0.0.0.0/0.0.0.0:33593] timeout=300.0>) 2021-05-03 03:52:09,789 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdee type=client open_count=1 channel=[id: 0x3c943039, 0.0.0.0/0.0.0.0:57509] timeout=300.0>) 2021-05-03 03:52:11,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdef type=client open_count=1 channel=[id: 0x3e7d27ad, 0.0.0.0/0.0.0.0:51247] timeout=300.0>) 2021-05-03 03:52:13,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdf0 type=client open_count=1 channel=[id: 0x0a815ee9, 0.0.0.0/0.0.0.0:49205] timeout=300.0>) 2021-05-03 03:52:15,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdf1 type=client open_count=1 channel=[id: 0x69164618, 0.0.0.0/0.0.0.0:49050] timeout=300.0>) 2021-05-03 03:52:17,888 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdf2 type=client open_count=1 channel=[id: 0x1f367aaf, 0.0.0.0/0.0.0.0:39015] timeout=300.0>) 2021-05-03 03:52:19,911 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdf3 type=client open_count=1 channel=[id: 0x204a9ead, 0.0.0.0/0.0.0.0:46824] timeout=300.0>) 2021-05-03 03:52:21,938 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5cdf4 type=client open_count=1 channel=[id: 0xc1edf449, 0.0.0.0/0.0.0.0:60351] timeout=300.0>) 2021-05-03 03:52:23,983 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdf5 type=client open_count=1 channel=[id: 0x4012193f, 0.0.0.0/0.0.0.0:50567] timeout=300.0>) 2021-05-03 03:52:31,892 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdf6 type=client open_count=1 channel=[id: 0x90fff8cd, 0.0.0.0/0.0.0.0:60019] timeout=300.0>) 2021-05-03 03:52:33,915 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdf7 type=client open_count=1 channel=[id: 0x0ec2cf80, 0.0.0.0/0.0.0.0:45719] timeout=300.0>) 2021-05-03 03:52:35,937 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdf8 type=client open_count=1 channel=[id: 0x23b672bf, 0.0.0.0/0.0.0.0:41329] timeout=300.0>) 2021-05-03 03:52:37,971 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdf9 type=client open_count=1 channel=[id: 0x08f34d56, 0.0.0.0/0.0.0.0:50439] timeout=300.0>) 2021-05-03 03:52:40,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdfa type=client open_count=1 channel=[id: 0x51daa9b5, 0.0.0.0/0.0.0.0:53893] timeout=300.0>) 2021-05-03 03:52:42,076 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdfb type=client open_count=1 channel=[id: 0x94e34efa, 0.0.0.0/0.0.0.0:49492] timeout=300.0>) 2021-05-03 03:52:44,101 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdfc type=client open_count=1 channel=[id: 0x3c649307, 0.0.0.0/0.0.0.0:37395] timeout=300.0>) 2021-05-03 03:52:46,118 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdfd type=client open_count=1 channel=[id: 0x43b173fd, 0.0.0.0/0.0.0.0:44238] timeout=300.0>) 2021-05-03 03:52:48,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5cdfe type=client open_count=1 channel=[id: 0x9cb0d5c3, 0.0.0.0/0.0.0.0:39925] timeout=300.0>) 2021-05-03 03:52:50,167 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5cdff type=client open_count=1 channel=[id: 0x43fe04c5, 0.0.0.0/0.0.0.0:59473] timeout=300.0>) 2021-05-03 03:53:03,924 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce00 type=client open_count=1 channel=[id: 0xb92cc423, 0.0.0.0/0.0.0.0:45784] timeout=300.0>) 2021-05-03 03:53:05,960 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce01 type=client open_count=1 channel=[id: 0x6fe8e7f1, 0.0.0.0/0.0.0.0:38303] timeout=300.0>) 2021-05-03 03:53:07,989 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce02 type=client open_count=1 channel=[id: 0x9400173d, 0.0.0.0/0.0.0.0:54714] timeout=300.0>) 2021-05-03 03:53:10,009 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce03 type=client open_count=1 channel=[id: 0x9844eb1d, 0.0.0.0/0.0.0.0:43596] timeout=300.0>) 2021-05-03 03:53:12,039 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce04 type=client open_count=1 channel=[id: 0x6ce2e9c1, 0.0.0.0/0.0.0.0:39107] timeout=300.0>) 2021-05-03 03:53:14,059 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce05 type=client open_count=1 channel=[id: 0x209b64f8, 0.0.0.0/0.0.0.0:45227] timeout=300.0>) 2021-05-03 03:53:16,095 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce06 type=client open_count=1 channel=[id: 0x32ca0c19, 0.0.0.0/0.0.0.0:35735] timeout=300.0>) 2021-05-03 03:53:18,128 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5ce07 type=client open_count=1 channel=[id: 0x09b3875c, 0.0.0.0/0.0.0.0:46951] timeout=300.0>) 2021-05-03 03:53:20,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5ce08 type=client open_count=1 channel=[id: 0x05f0753b, 0.0.0.0/0.0.0.0:49475] timeout=300.0>) 2021-05-03 03:53:25,588 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5ce09 type=client open_count=1 channel=[id: 0x0cdd36fc, 0.0.0.0/0.0.0.0:45770] timeout=300.0>) 2021-05-03 03:53:27,638 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5ce0a type=client open_count=1 channel=[id: 0x0503559e, 0.0.0.0/0.0.0.0:40924] timeout=300.0>) 2021-05-03 03:53:29,701 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5ce0b type=client open_count=1 channel=[id: 0x67707acb, 0.0.0.0/0.0.0.0:53513] timeout=300.0>) 2021-05-03 03:53:31,747 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5ce0c type=client open_count=1 channel=[id: 0x478348b6, 0.0.0.0/0.0.0.0:50656] timeout=300.0>) 2021-05-03 03:53:33,789 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5ce0d type=client open_count=1 channel=[id: 0x3e4b49de, 0.0.0.0/0.0.0.0:56315] timeout=300.0>) 2021-05-03 03:53:35,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 03:53:52,680 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 03:54:07,125 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 03:54:07,186 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 03:54:07,236 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 03:54:49,420 | test | INFO | pool-4-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 1.40951422099 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 03:54:49,552 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 03:54:59,694 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 03:54:59,733 | test | INFO | pool-4-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.3129999638 sec 2021-05-03 03:55:00,085 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 03:55:37,846 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 03:55:38,413 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 532383208 | 155129145 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 509206408 | 168605800 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 593736976 | 107494262 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 03:55:38,436 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 03:57:09,605 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 03:57:09,624 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 04:08:47,188 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **..........................**....................*......................................................*X | 65.984 | | 48.400 | ........................................................................*X | 48.400 | | 0.000 | *********************************************...*.............**......*X | 15.101 | | 8.000 | ...........**X | 8.000 | | 4.000 | .....**......**...*X | 10.498 | | 4.401 | .....*X | 4.401 | | 3.201 | ...**..X | 4.401 | | 4.400 | .....*X | 4.400 | | 2.800 | ...*X | 2.800 | | 0.000 | *************...*****..**........*.....**............X | 20.200 | | 15.800 | ......................*X | 15.800 | | 12.500 | .................X | 12.500 | | 6.100 | ........***.*X | 7.100 | | 3.900 | ....X | 3.900 | | 2.700 | ...*X | 2.700 | | 1.200 | X | 1.200 | | 0.400 | X | 0.400 | | 0.000 | ****************************.*..*X | 2.600 | | 2.600 | ..*X | 2.600 | | 0.400 | **X | 0.400 | | 0.000 | *****.*..*X | 2.600 | | 2.600 | ..*X | 2.600 | | 2.000 | ..X | 2.000 | | 0.400 | *X | 0.400 | | 0.000 | ******......*...................**................*X | 27.310 | | 27.309 | ........................................X | 27.309 | | 27.304 | ........................................*X | 27.304 | | 17.701 | .........................*X | 17.701 | | 5.600 | .......**..*X | 6.796 | | 6.196 | ........X | 6.196 | | 4.596 | .....*X | 4.596 | | 2.200 | ..*X | 2.200 | | 1.600 | .X | 1.600 | | 0.000 | **************************........***...**..**.**......*....................*........**X | 32.002 | | 26.602 | .......................................X | 26.602 | | 9.000 | ............X | 9.000 | | 0.600 | *X | 0.600 | | 0.000 | *.**.**.*X | 2.200 | | 1.400 | .*X | 1.400 | | 0.800 | X | 0.800 | | 0.000 | *****.*.*X | 1.500 | | 1.200 | X | 1.200 | | 0.400 | ****.*.X | 1.500 | | 1.100 | **.*.**.*X | 3.000 | | 2.400 | ..X | 2.400 | | 0.800 | *X | 0.800 | | 0.000 | **.**.....**......................**X | 19.095 | | 18.692 | ...........................X | 18.692 | | 14.792 | .....................X | 14.792 | | 5.399 | .......*X | 5.399 | | 2.601 | ..**.................***......X | 17.599 | | 13.997 | ....................X | 13.997 | | 4.400 | .....*X | 4.400 | | 0.000 | **********************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 04:08:47,618 | test | INFO | pool-4-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **..................**..............**.....................................X | 37.491 | | 27.500 | ..................................................*X | 27.500 | | 20.000 | ....................................X | 20.000 | | 0.000 | **************************************.**.*...*..**..................**......................................................*.....................*X | 53.803 | | 44.803 | ..................................................................................X | 44.803 | | 14.400 | .........................*X | 14.400 | | 1.200 | .*X | 1.200 | | 0.000 | ***********************..**.*X | 1.900 | | 1.900 | ..***.*.*......................................**................*............X | 38.703 | | 18.700 | .................................*X | 18.700 | | 8.700 | ...............*X | 8.700 | | 1.200 | .X | 1.200 | | 0.000 | **..**.........................................................**................*X | 40.799 | | 32.399 | ...........................................................X | 32.399 | | 10.000 | .................X | 10.000 | | 0.000 | ********************************************************************************************........*......................**...........*X | 22.505 | | 18.003 | ................................X | 18.003 | | 6.000 | ..........X | 6.000 | | 0.000 | ***********.*.....................**...........**.....X | 20.801 | | 10.400 | ..................X | 10.400 | | 4.400 | .......*X | 4.400 | | 0.800 | X | 0.800 | | 0.000 | ****.*.X | 1.500 | | 1.500 | .*X | 1.500 | | 1.500 | .**...*.**.X | 4.100 | | 2.500 | ...*X | 2.500 | | 2.500 | ...*X | 2.500 | | 1.900 | ..X | 1.900 | | 1.900 | ..X | 1.900 | | 1.100 | .*X | 1.100 | | 0.800 | X | 0.800 | | 0.000 | *****..**X | 1.500 | | 1.200 | .X | 1.200 | | 0.400 | X | 0.400 | | 0.000 | *****....**......*.............*.*X | 13.294 | | 10.894 | ...................X | 10.894 | | 4.999 | ........**.....**.....................................................**X | 36.401 | | 36.397 | ..................................................................X | 36.397 | | 27.994 | ...................................................X | 27.994 | | 5.600 | .........*X | 5.600 | | 0.000 | **********************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 04:08:47,789 | test | INFO | pool-4-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +----------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.........**...................*...............................................................**.........*X | 1457.445 | | 1177.445 | ...............................................................................**X | 1177.445 | | 1123.000 | ............................................................................X | 1123.000 | | 1101.638 | ..........................................................................X | 1101.638 | | 1072.538 | ........................................................................*X | 1072.538 | | 1054.438 | .......................................................................*X | 1054.438 | | 1021.957 | .....................................................................***X | 1030.157 | | 1025.795 | .....................................................................X | 1025.795 | | 993.195 | ...................................................................*X | 993.195 | | 955.995 | ................................................................*X | 955.995 | | 716.588 | ................................................**X | 716.588 | | 678.188 | .............................................X | 678.188 | | 672.959 | .............................................X | 672.959 | | 671.390 | .............................................****........***..X | 820.631 | | 809.514 | ......................................................*.**.**...****X | 881.907 | | 860.850 | ..........................................................*X | 860.850 | | 817.650 | .......................................................X | 817.650 | | 760.350 | ...................................................*X | 760.350 | | 588.635 | .......................................X | 588.635 | | 561.235 | .....................................**.*..**.*X | 624.451 | | 624.014 | .........................................**..**.*.......**..*..*...**..****X | 894.825 | | 858.225 | .........................................................X | 858.225 | | 793.737 | .....................................................X | 793.737 | | 620.400 | .........................................X | 620.400 | | 581.200 | ......................................**.*...**.**.*....*..**X | 756.843 | | 726.358 | ................................................*X | 726.358 | | 689.258 | ..............................................**.*...*...**X | 801.568 | | 776.230 | ....................................................*X | 776.230 | | 726.030 | ................................................*X | 726.030 | | 660.430 | ............................................X | 660.430 | | 533.259 | ...................................X | 533.259 | | 526.459 | ...................................**.*....**..*...*............**...*.X | 903.800 | | 864.828 | ..........................................................*X | 864.828 | | 852.428 | .........................................................**.X | 860.328 | | 833.775 | ........................................................X | 833.775 | | 822.275 | .......................................................*X | 822.275 | | 795.475 | .....................................................X | 795.475 | | 786.881 | ....................................................*X | 786.881 | | 769.881 | ...................................................X | 769.881 | | 740.181 | .................................................X | 740.181 | | 606.798 | ........................................*X | 606.798 | | 579.398 | ......................................*X | 579.398 | | 574.198 | ......................................*.***.X | 602.859 | | 566.527 | .....................................**.*.*...........**...**....*..**X | 883.682 | | 874.982 | ...........................................................**X | 877.100 | | 857.338 | .........................................................*X | 857.338 | | 804.438 | ......................................................X | 804.438 | | 620.438 | .........................................X | 620.438 | | 578.645 | ......................................***.**.***.**.**..*.......*.**.*....**.*X | 862.681 | | 822.564 | .......................................................*X | 822.564 | | 798.937 | .....................................................*X | 798.937 | | 743.901 | ..................................................X | 743.901 | | 609.782 | ........................................*X | 609.782 | | 589.304 | .......................................*.*....**.**..**.*X | 725.759 | | 718.474 | ................................................*X | 728.171 | | 725.071 | ................................................**..*.***X | 766.290 | | 756.593 | ..................................................*..*X | 785.000 | | 771.543 | ...................................................X | 771.543 | | 727.700 | ................................................X | 727.700 | | 595.000 | .......................................*X | 595.000 | | 562.201 | .....................................**.**.*X | 583.712 | | 575.665 | ......................................X | 575.665 | | 543.833 | ....................................****..*..........**..**..*.**X | 800.550 | | 792.048 | .....................................................X | 792.048 | | 770.648 | ...................................................***X | 771.217 | | 736.069 | .................................................X | 736.069 | | 666.754 | ............................................X | 666.754 | | 626.108 | .........................................*X | 626.108 | | 625.808 | .........................................X | 625.808 | | 624.340 | .........................................**.**.*.**.X | 676.882 | | 667.682 | ............................................*...**.**.......***..*..*.*X | 897.721 | | 884.452 | ...........................................................*X | 884.452 | | 838.552 | ........................................................X | 838.552 | | 742.931 | .................................................X | 742.931 | | 675.642 | .............................................**X | 675.642 | | 657.242 | ............................................X | 657.242 | | 553.485 | ....................................*..*X | 576.166 | | 573.718 | ......................................*....X | 628.461 | | 625.913 | .........................................***.****...*.........*..*X | 838.117 | | 827.517 | .......................................................X | 827.517 | | 790.691 | .....................................................*X | 790.691 | | 786.591 | ....................................................**.X | 788.891 | | 712.407 | ...............................................X | 712.407 | | 621.807 | .........................................*X | 621.807 | | 477.007 | ...............................*X | 477.007 | | 86.398 | ....*X | 86.398 | | 13.398 | X | 13.398 | | 1.698 | X | 1.698 | | 0.000 | ******....X | 62.562 | +----------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 04:08:48,094 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 27.5720164609 | 3693387776 | 2572083200 | 0 / 3909087232 | 100374 / 205756 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 31.132802466 | 3693387776 | 2614558720 | 0 / 3909087232 | 101999 / 203171 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 22.4054808424 | 6078930944 | 4837912576 | 0 / 3758092288 | 100686 / 197453 | 7.0.0-5085-enterprise | | 172.23.120.206 | kv | 13.0952380952 | 3691520000 | 2774405120 | 0 / 0 | 99394 / 199451 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 31.5897435897 | 4201836544 | 3170852864 | 0 / 3758092288 | 99772 / 198778 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 04:09:01,996 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 04:11:10,246 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 157975176 | 247116351 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 158296392 | 275896507 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 437352440 | 476094231 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-03 04:11:10,247 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 04:11:21,490 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_out 2021-05-03 04:11:21,492 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_out 2021-05-03 04:11:21,582 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 04:11:21,953 | test | INFO | pool-4-thread-26 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 15.4454940005 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 11.9059785386 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 10.990945674 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 6.65149215984 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 8.32907511497 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 04:11:21,993 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 04:11:29,375 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 04:11:32,035 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 04:11:37,201 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 4.17 2021-05-03 04:11:42,355 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 7.21 2021-05-03 04:11:44,153 | test | INFO | pool-4-thread-22 | [table_view:display:72] Keys failed in bucket1:_default:LDCmqxKNt-0-253000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:11:44,153 | test | INFO | pool-4-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:9pKED3zw0LQ_p7gSVQtC-0-254000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:11:44,153 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket1:_default:E8%Q438hQR-0-256000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:11:47,493 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 10.36 2021-05-03 04:11:48,542 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 04:11:48,543 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 04:11:52,767 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 12.73 2021-05-03 04:11:57,306 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 04:11:57,980 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 15.1 2021-05-03 04:12:03,144 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 17.56 2021-05-03 04:12:08,338 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 19.78 2021-05-03 04:12:12,726 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in bucket1:bV-27-143000:NU6TnlUKq9%XovVz7-27-300000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-313 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-283 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:12,727 | test | INFO | pool-4-thread-12 | [table_view:display:72] Keys failed in bucket1:bV-27-143000:NU6TnlUKq9%XovVz7-27-300000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-313 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-283 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:12,730 | test | INFO | pool-4-thread-28 | [table_view:display:72] Keys failed in bucket1:9tKldt-26-710000:8xmfc0%QRiw9Km3W3-26-855000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-313 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-283 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:12,732 | test | INFO | pool-4-thread-28 | [table_view:display:72] Keys failed in bucket1:9tKldt-26-710000:8xmfc0%QRiw9Km3W3-26-855000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-313 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-283 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:13,499 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 22.28 2021-05-03 04:12:13,576 | test | INFO | pool-4-thread-10 | [table_view:display:72] Keys failed in bucket1:bV-27-143000:3ERH-27-301000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-85 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:13,578 | test | INFO | pool-4-thread-10 | [table_view:display:72] Keys failed in bucket1:bV-27-143000:3ERH-27-301000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-85 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:18,667 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 24.55 2021-05-03 04:12:23,782 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 26.88 2021-05-03 04:12:28,957 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 29.11 2021-05-03 04:12:34,082 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 31.87 2021-05-03 04:12:38,874 | test | INFO | pool-4-thread-10 | [table_view:display:72] Keys failed in bucket1:bV-27-143000:3ERH-27-301000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-437 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:38,875 | test | INFO | pool-4-thread-10 | [table_view:display:72] Keys failed in bucket1:bV-27-143000:3ERH-27-301000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-437 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-457 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:12:39,191 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 04:12:44,280 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 04:12:49,417 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 04:12:54,615 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 35.68 2021-05-03 04:12:59,821 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 39.11 2021-05-03 04:13:03,551 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in bucket2:_default:jHLTD_ChO-0-221000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:03,553 | test | INFO | pool-4-thread-19 | [table_view:display:72] Keys failed in bucket2:Z5nFu5-0-245000:4W-yYT_z%WqtYH9q7tMt-0-249000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:03,556 | test | INFO | pool-4-thread-14 | [table_view:display:72] Keys failed in bucket2:_default:-GoWI90s-0-223000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:03,644 | test | INFO | pool-4-thread-1 | [table_view:display:72] Keys failed in bucket2:kLJIUscF1_r-26-317000:vxaslEClNsG-26-511000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:03,644 | test | INFO | pool-4-thread-1 | [table_view:display:72] Keys failed in bucket2:kLJIUscF1_r-26-317000:vxaslEClNsG-26-511000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2132 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:05,017 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 42.29 2021-05-03 04:13:10,227 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 44.58 2021-05-03 04:13:12,240 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1810 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1980 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:12,240 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1810 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1980 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:15,332 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 47.05 2021-05-03 04:13:20,625 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 48.86 2021-05-03 04:13:25,762 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 51.21 2021-05-03 04:13:27,457 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:27,460 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2171 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:30,642 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:b3n8vwWOKLSh-26-147000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1648 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:30,644 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:b3n8vwWOKLSh-26-147000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1648 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:30,894 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 53.3 2021-05-03 04:13:36,017 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 55.38 2021-05-03 04:13:41,204 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 57.1 2021-05-03 04:13:43,111 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2219 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2389 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:43,111 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2219 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2389 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:46,397 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 59.42 2021-05-03 04:13:48,352 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:b3n8vwWOKLSh-26-147000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1972 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:48,354 | test | INFO | pool-4-thread-2 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:b3n8vwWOKLSh-26-147000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1947 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1972 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:51,519 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 61.25 2021-05-03 04:13:55,586 | test | INFO | pool-4-thread-9 | [table_view:display:72] Keys failed in bucket2:1VWkDzXAIVvBXF%X%l-25-311000:GvHrsBSZkvFIlv-25-430000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2850 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:55,586 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in bucket2:1VWkDzXAIVvBXF%X%l-25-311000:ln9-IojWKM%osoq_IlbQ-25-429000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2850 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:55,588 | test | INFO | pool-4-thread-9 | [table_view:display:72] Keys failed in bucket2:1VWkDzXAIVvBXF%X%l-25-311000:GvHrsBSZkvFIlv-25-430000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2850 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:55,588 | test | INFO | pool-4-thread-27 | [table_view:display:72] Keys failed in bucket2:1VWkDzXAIVvBXF%X%l-25-311000:ln9-IojWKM%osoq_IlbQ-25-429000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2850 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:55,598 | test | INFO | pool-4-thread-23 | [table_view:display:72] Keys failed in bucket2:kLJIUscF1_r-26-317000:UfE4L0Ez3Ka%G8C-26-510000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2850 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:55,598 | test | INFO | pool-4-thread-23 | [table_view:display:72] Keys failed in bucket2:kLJIUscF1_r-26-317000:UfE4L0Ez3Ka%G8C-26-510000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2850 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:13:56,668 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 63.38 2021-05-03 04:14:01,828 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 65.4 2021-05-03 04:14:07,019 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 04:14:08,130 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:08,131 | test | INFO | pool-4-thread-30 | [table_view:display:72] Keys failed in bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2426 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:12,171 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 04:14:19,979 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.91 2021-05-03 04:14:24,082 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:_default:HXU0-YufiN-0-283000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-258 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:25,099 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 71.71 2021-05-03 04:14:26,697 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:_default:HXU0-YufiN-0-283000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-836 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-873 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:30,279 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 74.74 2021-05-03 04:14:35,420 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 77.08 2021-05-03 04:14:40,523 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 78.64 2021-05-03 04:14:42,536 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:_default:HXU0-YufiN-0-283000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-426 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-463 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:42,737 | test | INFO | pool-4-thread-19 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:42,739 | test | INFO | pool-4-thread-19 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2584 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:43,684 | test | INFO | pool-4-thread-11 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-61 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-136 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-125 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-178 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-173 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-160 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:45,812 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 80.64 2021-05-03 04:14:47,756 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:_default:HXU0-YufiN-0-283000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-969 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:48,868 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-101 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-160 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:50,937 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 82.34 2021-05-03 04:14:54,286 | test | INFO | pool-4-thread-29 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-601 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-612 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-644 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:54,579 | test | INFO | pool-4-thread-4 | [table_view:display:72] Keys failed in default:L4-27-823000:MHNGdBEFiso0bCd-28-17000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1048 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:54,581 | test | INFO | pool-4-thread-4 | [table_view:display:72] Keys failed in default:L4-27-823000:MHNGdBEFiso0bCd-28-17000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1048 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:54,579 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in default:L4-27-823000:Uh3ADBjTehP9u-27-986000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1048 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:54,582 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in default:L4-27-823000:cUHwFNZXvHl8pVuzY-29-163000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1048 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:54,582 | test | INFO | pool-4-thread-13 | [table_view:display:72] Keys failed in default:L4-27-823000:Uh3ADBjTehP9u-27-986000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1048 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:54,582 | test | INFO | pool-4-thread-21 | [table_view:display:72] Keys failed in default:L4-27-823000:cUHwFNZXvHl8pVuzY-29-163000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1048 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:14:56,048 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 83.93 2021-05-03 04:15:01,068 | test | INFO | pool-4-thread-14 | [table_view:display:72] Keys failed in default:_default:F-TqNZo-0-286000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-48 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-114 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-111 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-82 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-154 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:01,161 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 85.67 2021-05-03 04:15:04,240 | test | INFO | pool-4-thread-19 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:04,242 | test | INFO | pool-4-thread-19 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2767 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:06,305 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 87.07 2021-05-03 04:15:09,368 | test | INFO | pool-4-thread-11 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-338 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-245 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-256 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-267 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-378 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-222 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:11,484 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 88.74 2021-05-03 04:15:12,269 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:xXKqta-0-302000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-537 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-515 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-666 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-589 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-663 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-550 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:12,753 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-338 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:16,596 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 90.72 2021-05-03 04:15:18,164 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:xXKqta-0-302000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-87 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-113 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-114 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-91 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-166 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:21,647 | test | INFO | pool-4-thread-29 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-809 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-768 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-779 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-886 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-881 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-890 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:21,766 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 92.31 2021-05-03 04:15:26,053 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:26,055 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2755 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:26,867 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 94.29 2021-05-03 04:15:28,947 | test | INFO | pool-4-thread-14 | [table_view:display:72] Keys failed in default:_default:F-TqNZo-0-286000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-338 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-231 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-271 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:31,986 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 96.0 2021-05-03 04:15:37,062 | test | INFO | pool-4-thread-6 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:QoV_-0-305000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-635 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-503 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-650 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:37,131 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 98.03 2021-05-03 04:15:40,265 | test | INFO | pool-4-thread-11 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-459 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-493 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:42,223 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 99.12 2021-05-03 04:15:42,588 | test | INFO | pool-4-thread-16 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-493 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:47,306 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 04:15:49,440 | test | INFO | pool-4-thread-29 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-913 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-956 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:52,388 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 04:15:54,460 | test | INFO | pool-4-thread-24 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:xXKqta-0-302000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-849 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:15:58,380 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 04:16:00,980 | test | INFO | pool-4-thread-7 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:QoV_-0-305000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-39 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:16:01,592 | test | INFO | pool-4-thread-8 | [table_view:display:72] Keys failed in default:3ADtwdSt74pdAWQ9R-0-301000:xXKqta-0-302000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-234 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-271 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:16:09,528 | test | INFO | pool-4-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 04:16:09,898 | test | INFO | pool-4-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 287.944999933 sec 2021-05-03 04:17:09,898 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 04:17:09,908 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 04:17:09,911 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 04:17:09,911 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 04:17:09,913 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 04:17:09,914 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 04:17:09,917 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-03 04:17:09,917 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 3 docs from 0 to 5000 2021-05-03 04:17:09,920 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 3 docs from 0 to 5000 2021-05-03 04:17:09,928 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 04:17:09,930 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 04:17:09,930 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 3 2021-05-03 04:19:54,911 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 04:20:39,522 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 04:20:39,523 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 04:20:39,525 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-03 04:20:39,526 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 5 docs from 0 to 500 2021-05-03 04:20:39,529 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 15 docs from 0 to 500 2021-05-03 04:20:39,529 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 04:20:39,530 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 17 docs from 500 to 980 2021-05-03 04:20:39,532 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 04:20:39,532 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 04:20:39,533 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-03 04:20:39,535 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 04:20:39,536 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-03 04:20:39,536 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 04:20:39,536 | test | ERROR | pool-4-thread-20 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 04:29:04,285 | test | INFO | pool-4-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.*..*....**........*X | 33.708 | | 33.408 | ..............*..X | 37.600 | | 32.700 | ..............X | 32.700 | | 26.300 | ...........**..........**.................*....*..*X | 99.165 | | 89.391 | ........................................*X | 89.531 | | 84.431 | ......................................**X | 84.820 | | 83.309 | .....................................*...**.......X | 104.109 | | 99.900 | .............................................*.X | 103.800 | | 96.300 | ...........................................**...X | 102.500 | | 95.400 | ...........................................X | 95.400 | | 75.200 | .................................*X | 75.200 | | 55.197 | ........................**.........X | 74.290 | | 71.671 | ................................*.............*X | 100.683 | | 94.195 | ..........................................*X | 94.195 | | 82.334 | .....................................**...............*.........*.........X | 154.600 | | 108.200 | ................................................*X | 108.200 | | 82.280 | .....................................**............*X | 109.739 | | 91.770 | .........................................X | 91.770 | | 83.591 | .....................................**......*.....*X | 107.868 | | 102.331 | ..............................................*..*...**......X | 125.814 | | 92.091 | .........................................*X | 92.091 | | 79.191 | ...................................**.X | 80.910 | | 70.269 | ...............................*.......*............*.....*X | 121.827 | | 86.352 | ......................................X | 86.352 | | 66.905 | .............................**...................**..**.*............*..X | 143.400 | | 120.282 | ......................................................*X | 120.282 | | 96.382 | ...........................................X | 96.382 | | 78.982 | ...................................*.............**........*X | 125.138 | | 85.897 | ......................................*X | 85.897 | | 60.697 | ...........................*..............*...................**............X | 158.511 | | 149.811 | ....................................................................X | 149.811 | | 146.500 | ..................................................................*X | 146.500 | | 145.500 | ..................................................................X | 145.500 | | 132.800 | ............................................................*X | 132.800 | | 129.200 | ..........................................................X | 129.200 | | 123.200 | .......................................................*.....*X | 133.291 | | 92.291 | .........................................*X | 92.291 | | 89.908 | ........................................***..................................*..*...........*X | 190.800 | | 165.583 | ...........................................................................**........................*X | 216.454 | | 209.986 | ................................................................................................*X | 209.986 | | 168.086 | ............................................................................X | 168.086 | | 120.625 | ......................................................*X | 120.625 | | 114.401 | ...................................................*X | 114.401 | | 81.501 | ....................................*.............*X | 108.491 | | 79.303 | ...................................*X | 79.303 | | 42.084 | ..................***...*....*.X | 59.513 | | 52.113 | .......................*X | 52.113 | | 43.402 | ...................**X | 43.402 | | 43.202 | ..................X | 43.202 | | 36.798 | ................X | 36.798 | | 28.002 | ...........*X | 28.002 | | 16.302 | ......*X | 16.302 | | 10.206 | ...X | 10.206 | | 4.401 | .X | 4.401 | | 3.301 | **..X | 6.700 | | 6.000 | .**X | 6.300 | | 6.300 | .X | 6.300 | | 3.900 | **X | 3.900 | | 3.700 | X | 3.700 | | 1.900 | *X | 1.900 | | 1.700 | X | 1.700 | | 1.600 | **.****X | 4.200 | | 1.400 | **X | 1.400 | | 0.700 | *X | 0.700 | | 0.400 | ****.*X | 2.600 | | 2.100 | *X | 2.100 | | 0.500 | X | 0.500 | | 0.100 | *****X | 2.001 | | 2.001 | *X | 2.001 | | 2.000 | X | 2.000 | | 1.900 | **X | 2.000 | | 1.000 | X | 1.000 | | 0.700 | *X | 0.700 | | 0.200 | ****.*X | 2.599 | | 1.401 | X | 1.401 | | 1.001 | *X | 1.001 | | 0.200 | ******.**X | 2.300 | | 2.000 | *X | 2.000 | | 1.600 | ****X | 1.900 | | 0.500 | **X | 0.500 | | 0.000 | *********X | 1.301 | | 0.500 | **X | 0.600 | | 0.600 | *X | 0.600 | | 0.400 | X | 0.400 | | 0.300 | *X | 0.300 | | 0.200 | *X | 0.200 | | 0.100 | *X | 0.100 | | 0.100 | *********X | 0.900 | | 0.500 | *X | 0.700 | | 0.400 | *******X | 0.900 | | 0.700 | X | 0.700 | | 0.500 | *X | 0.500 | | 0.200 | *X | 0.200 | | 0.000 | ****X | 0.600 | | 0.400 | *X | 0.400 | | 0.200 | *****X | 0.500 | | 0.100 | **X | 0.100 | | 0.000 | ********X | 0.600 | | 0.400 | *X | 0.400 | | 0.200 | ****X | 0.300 | | 0.300 | **X | 0.300 | | 0.200 | *****X | 0.200 | | 0.000 | *********X | 0.200 | | 0.100 | *X | 0.100 | | 0.000 | **********X | 0.300 | | 0.100 | *X | 0.200 | | 0.200 | **X | 0.400 | | 0.400 | ***X | 0.500 | | 0.300 | X | 0.300 | | 0.200 | *X | 0.200 | | 0.000 | *****************X | 0.200 | | 0.100 | **X | 0.100 | | 0.000 | *****X | 0.200 | | 0.100 | **X | 0.300 | | 0.200 | **X | 0.200 | | 0.000 | **********X | 0.500 | | 0.500 | X | 0.500 | | 0.400 | *X | 0.400 | | 0.400 | X | 0.400 | | 0.300 | ***X | 0.300 | | 0.100 | *X | 0.100 | | 0.000 | ********X | 0.600 | | 0.400 | *X | 0.400 | | 0.200 | X | 0.200 | | 0.000 | *****************X | 0.200 | | 0.100 | ********X | 0.600 | | 0.500 | X | 0.500 | | 0.400 | **X | 0.400 | | 0.100 | X | 0.100 | +---------+---------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 04:29:05,325 | test | INFO | pool-4-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.*.***...........**.......*.............*...................................*.............**.........*X | 198.891 | | 191.575 | ......................................................................................**........*X | 210.451 | | 202.279 | ...........................................................................................X | 202.279 | | 171.627 | .............................................................................X | 171.627 | | 163.627 | .........................................................................**....*..........*...**...*X | 207.414 | | 197.600 | ........................................................................................*...........X | 219.800 | | 198.100 | .........................................................................................*X | 198.100 | | 111.600 | .................................................**......*.X | 125.900 | | 119.396 | .....................................................*X | 119.396 | | 114.389 | ...................................................X | 114.389 | | 94.279 | .........................................X | 94.279 | | 71.087 | ...............................X | 71.087 | | 68.099 | .............................**............X | 94.331 | | 85.125 | .....................................**.X | 85.800 | | 78.399 | ..................................*.....X | 88.899 | | 73.699 | ................................X | 73.699 | | 63.894 | ............................*X | 63.894 | | 48.380 | .....................*...*.*...*X | 65.897 | | 57.592 | .........................X | 57.592 | | 48.083 | ....................*X | 48.083 | | 47.728 | ....................X | 47.728 | | 46.438 | ....................*........**..X | 69.505 | | 68.785 | ..............................*X | 68.785 | | 57.785 | .........................**.........**.........X | 97.045 | | 78.326 | ..................................*..*X | 82.441 | | 79.375 | ...................................*...*X | 86.218 | | 67.902 | .............................***..*..........X | 93.805 | | 74.105 | ................................*......***........X | 105.274 | | 73.874 | ................................X | 73.874 | | 50.109 | .....................*X | 50.109 | | 25.709 | ..........*.............*.........**...............*X | 106.294 | | 81.594 | ....................................*..*X | 86.007 | | 78.107 | ..................................*............*.**.*........X | 125.300 | | 92.700 | .........................................**..X | 97.300 | | 68.195 | ..............................**................*X | 104.795 | | 89.807 | .......................................**X | 89.807 | | 61.107 | ..........................X | 61.107 | | 55.207 | ........................X | 55.207 | | 35.000 | ..............**...**......*..*X | 57.387 | | 43.595 | ..................X | 43.595 | | 35.395 | ...............X | 35.395 | | 27.906 | ...........***X | 28.502 | | 18.502 | .......X | 18.502 | | 15.499 | ......*X | 15.499 | | 8.400 | ..X | 8.400 | | 6.400 | .X | 6.400 | | 0.000 | ******X | 0.400 | | 0.200 | X | 0.200 | | 0.100 | *X | 0.100 | | 0.000 | ********X | 1.400 | | 1.100 | **.***.***X | 5.000 | | 3.100 | *X | 3.100 | | 0.000 | ******X | 1.600 | | 1.400 | *X | 1.400 | | 0.000 | ************X | 1.200 | | 0.400 | *****X | 1.201 | | 1.001 | *X | 1.001 | | 0.601 | *X | 0.601 | | 0.100 | ****X | 1.100 | | 1.100 | **.*.**X | 6.495 | | 6.297 | .*X | 6.297 | | 2.701 | X | 2.701 | | 1.401 | X | 1.401 | | 0.500 | *X | 0.500 | | 0.300 | **X | 0.600 | | 0.300 | *****X | 1.100 | | 0.900 | **X | 1.000 | | 0.900 | *X | 0.900 | | 0.100 | X | 0.100 | | 0.000 | **********X | 1.301 | | 1.201 | *X | 1.201 | | 0.200 | ****X | 1.500 | | 1.400 | *X | 1.400 | | 1.100 | *X | 1.100 | | 0.700 | ******X | 1.300 | | 1.100 | **X | 1.100 | | 1.000 | *X | 1.000 | | 0.800 | *X | 0.800 | | 0.600 | ***X | 1.000 | | 0.400 | X | 0.400 | | 0.200 | *X | 0.200 | | 0.000 | **********X | 0.400 | | 0.200 | ********X | 0.700 | | 0.300 | X | 0.300 | | 0.200 | *X | 0.200 | | 0.000 | ********X | 1.300 | | 1.000 | **X | 1.400 | | 1.000 | **X | 1.500 | | 0.900 | X | 0.900 | | 0.900 | *X | 0.900 | | 0.400 | *X | 0.400 | | 0.200 | *X | 0.200 | | 0.200 | **X | 0.400 | | 0.300 | *X | 0.300 | | 0.200 | ********X | 1.300 | | 1.000 | *X | 1.000 | | 0.200 | X | 0.200 | | 0.100 | *X | 0.100 | | 0.000 | **********X | 0.700 | | 0.700 | X | 0.700 | | 0.600 | X | 0.600 | | 0.400 | *X | 0.400 | | 0.200 | X | 0.200 | | 0.100 | *X | 0.100 | | 0.000 | **********X | 0.200 | | 0.000 | ******X | 0.300 | | 0.200 | *X | 0.200 | | 0.000 | ******X | 0.600 | | 0.400 | **X | 0.600 | | 0.300 | *X | 0.300 | | 0.200 | *X | 0.200 | | 0.000 | **********X | 0.800 | | 0.500 | *X | 0.500 | | 0.100 | X | 0.100 | | 0.000 | ********X | 0.400 | | 0.200 | ******X | 1.000 | | 0.700 | X | 0.700 | | 0.300 | **X | 0.300 | | 0.200 | *X | 0.200 | | 0.000 | ********X | 0.700 | | 0.600 | *X | 0.600 | | 0.400 | *****X | 0.500 | | 0.300 | X | 0.300 | | 0.300 | *****X | 0.600 | | 0.400 | **X | 0.400 | | 0.100 | X | 0.100 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 04:29:07,288 | test | INFO | pool-4-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******..*......*....**..*X | 128.800 | | 108.300 | ...........X | 108.300 | | 88.499 | .........X | 88.499 | | 28.199 | ..*X | 28.199 | | 14.199 | **.*..**X | 37.707 | | 21.507 | .X | 21.507 | | 11.403 | **.......*......................**..............*X | 391.200 | | 362.200 | .......................................*X | 362.200 | | 276.500 | ..............................X | 276.500 | | 144.400 | ...............**...*.....*............*X | 326.486 | | 318.875 | ...................................*..*.......*.*X | 408.059 | | 207.841 | ......................X | 207.841 | | 173.329 | ..................*X | 173.329 | | 169.900 | ..................X | 169.900 | | 135.592 | ..............***...**X | 166.659 | | 156.359 | ................X | 156.359 | | 108.275 | ...........*X | 108.275 | | 90.100 | .........*.*...**..X | 148.456 | | 144.472 | ...............*X | 144.472 | | 123.508 | ............**.X | 124.486 | | 97.986 | ..........*...*X | 123.986 | | 107.878 | ...........X | 107.878 | | 102.358 | ..........**..*..X | 138.526 | | 117.555 | ............**X | 118.872 | | 105.813 | ..........*...**.**..X | 156.008 | | 137.208 | ..............X | 137.208 | | 104.900 | ..........X | 104.900 | | 77.778 | .......**..*X | 88.878 | | 67.907 | ......*X | 67.907 | | 32.507 | ..**.**..**..**..*X | 91.807 | | 72.700 | .......**...*..**.......*X | 184.200 | | 173.000 | ..................X | 173.000 | | 162.292 | .................X | 162.292 | | 119.592 | ............**......*...**....*X | 237.613 | | 139.700 | ..............**X | 139.700 | | 127.181 | .............*...........*.**...*X | 262.003 | | 239.503 | ..........................*.X | 251.458 | | 207.820 | ......................**X | 207.820 | | 153.620 | ................X | 153.620 | | 133.690 | ..............**....*...........*......*X | 322.145 | | 286.127 | ...............................*X | 286.127 | | 223.459 | ........................*.......*X | 283.759 | | 259.904 | ............................**X | 259.904 | | 235.804 | .........................*......*.................**...........................*............X | 779.157 | | 779.157 | .......................................................................................*...........X | 876.203 | | 728.603 | .................................................................................*X | 728.603 | | 522.100 | ..........................................................***...........*...........*...............*X | 853.400 | | 802.400 | .........................................................................................*X | 802.400 | | 769.400 | ......................................................................................X | 769.400 | | 668.500 | ..........................................................................X | 668.500 | | 575.900 | ................................................................*X | 575.900 | | 505.100 | ........................................................*..**..............**...**X | 679.132 | | 564.232 | ..............................................................X | 564.232 | | 507.951 | ........................................................X | 507.951 | | 485.151 | .....................................................**....**............X | 623.100 | | 623.100 | .....................................................................**..*........*..**........*X | 803.884 | | 561.418 | ..............................................................*X | 561.418 | | 453.083 | ..................................................**................*.............**....................X | 883.979 | | 864.579 | ................................................................................................X | 864.579 | | 788.900 | ........................................................................................*X | 788.900 | | 663.991 | ..........................................................................X | 663.991 | | 643.545 | .......................................................................**........*...*X | 736.887 | | 734.676 | ..................................................................................X | 734.676 | | 651.176 | ........................................................................X | 651.176 | | 647.943 | ........................................................................*X | 647.943 | | 556.500 | .............................................................**....X | 584.500 | | 576.800 | ................................................................**.................X | 725.400 | | 725.400 | .................................................................................X | 725.400 | | 706.000 | ..............................................................................*X | 706.000 | | 665.300 | ..........................................................................X | 665.300 | | 558.470 | ..............................................................*X | 558.470 | | 510.570 | ........................................................*X | 510.570 | | 447.432 | .................................................**X | 447.432 | | 388.874 | ..........................................*..X | 400.857 | | 396.425 | ...........................................**............X | 495.199 | | 495.199 | .......................................................*...............**..**.*.......**........*X | 787.655 | | 769.955 | ......................................................................................X | 769.955 | | 678.673 | ...........................................................................X | 678.673 | | 676.780 | ...........................................................................**.........**.............X | 869.153 | | 834.800 | .............................................................................................*X | 834.800 | | 498.754 | .......................................................*X | 498.754 | | 381.954 | ..........................................*X | 381.954 | | 333.831 | ....................................*..**...**.........**.*X | 461.700 | | 426.400 | ...............................................X | 426.400 | | 419.400 | ..............................................*X | 419.400 | | 392.900 | ...........................................*X | 392.900 | | 353.042 | ......................................**.**...*......**........**.....*X | 555.300 | | 499.600 | .......................................................*X | 499.600 | | 477.900 | .....................................................X | 477.900 | | 477.000 | ....................................................**........*X | 548.000 | | 512.400 | ........................................................X | 512.400 | | 486.700 | ......................................................X | 486.700 | | 382.500 | ..........................................*X | 382.500 | | 373.600 | .........................................X | 373.600 | | 352.286 | ......................................**..*.*..**.*X | 401.869 | | 341.869 | .....................................X | 341.869 | | 310.785 | ..................................*X | 310.785 | | 287.166 | ...............................*.....*.**....*X | 373.319 | | 361.914 | .......................................*....X | 391.514 | | 368.345 | ........................................**.....X | 408.000 | | 383.200 | ..........................................**.*X | 395.600 | | 346.800 | ......................................*.X | 360.600 | | 355.600 | .......................................**....X | 393.000 | | 372.500 | .........................................**.*X | 387.800 | | 369.300 | ........................................X | 369.300 | | 366.300 | ........................................X | 366.300 | | 309.600 | ..................................*X | 309.600 | | 277.682 | ..............................X | 277.682 | | 267.082 | .............................**.*X | 282.282 | | 273.318 | .............................*X | 273.318 | | 266.500 | .............................**.X | 275.382 | | 255.555 | ...........................*X | 255.555 | | 241.187 | ..........................*X | 241.187 | | 222.740 | ........................*....*.......X | 320.496 | | 320.496 | ...................................*...X | 349.208 | | 341.264 | .....................................X | 341.264 | | 296.064 | ................................**..*..**......**X | 383.200 | | 362.113 | .......................................X | 362.113 | | 333.991 | ....................................**.*....X | 373.743 | | 373.743 | .........................................*.*X | 388.591 | | 354.791 | .......................................X | 354.791 | | 315.062 | ..................................*X | 315.062 | | 263.836 | ............................*X | 263.836 | | 231.473 | .........................**X | 231.473 | | 216.173 | .......................X | 216.173 | | 190.173 | ....................*.**....**......*.*..*X | 311.865 | | 284.847 | ...............................*..*X | 303.848 | | 288.548 | ...............................**...X | 316.089 | | 304.868 | .................................**..*X | 320.868 | | 315.345 | ..................................*...*..**X | 361.647 | | 361.024 | .......................................*X | 361.024 | | 334.224 | ....................................**.X | 341.824 | | 325.778 | ...................................*...*X | 352.578 | | 330.278 | ....................................X | 330.278 | | 307.695 | .................................*X | 307.695 | | 283.395 | ...............................*..*...*X | 329.932 | | 320.332 | ...................................X | 320.332 | | 289.150 | ...............................*X | 289.150 | | 281.419 | ..............................*.**....X | 318.370 | | 307.027 | .................................*..*X | 320.327 | | 314.727 | ..................................*...**..**....**X | 392.800 | | 390.600 | ...........................................**.**...**.....*.......*........**...X | 629.953 | | 606.316 | ...................................................................*X | 606.316 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 04:29:07,505 | test | WARNING | Thread-5262 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:LDCmqxKNt-0-253000 delete failed keys from task: [u'test_collections-1'] 2021-05-03 04:29:07,530 | test | WARNING | Thread-5264 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:E8%Q438hQR-0-256000 delete failed keys from task: [u'test_collections-1'] 2021-05-03 04:29:07,536 | test | WARNING | Thread-5270 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:9pKED3zw0LQ_p7gSVQtC-0-254000 delete failed keys from task: [u'test_collections-1'] 2021-05-03 04:29:07,568 | test | WARNING | Thread-5332 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:bV-27-143000:3ERH-27-301000 create failed keys from task: [u'test_collections-437', u'test_collections-85', u'test_collections-457'] 2021-05-03 04:29:07,572 | test | WARNING | Thread-5334 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:9tKldt-26-710000:8xmfc0%QRiw9Km3W3-26-855000 create failed keys from task: [u'test_collections-313', u'test_collections-283'] 2021-05-03 04:29:07,568 | test | WARNING | Thread-5333 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:bV-27-143000:NU6TnlUKq9%XovVz7-27-300000 create failed keys from task: [u'test_collections-313', u'test_collections-283'] 2021-05-03 04:29:07,592 | test | WARNING | Thread-5428 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:-GoWI90s-0-223000 delete failed keys from task: [u'test_collections-1'] 2021-05-03 04:29:07,594 | test | WARNING | Thread-5431 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:jHLTD_ChO-0-221000 delete failed keys from task: [u'test_collections-1'] 2021-05-03 04:29:07,628 | test | WARNING | Thread-5487 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:1VWkDzXAIVvBXF%X%l-25-311000:ln9-IojWKM%osoq_IlbQ-25-429000 create failed keys from task: [u'test_collections-2850'] 2021-05-03 04:29:07,632 | test | WARNING | Thread-5488 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:ialDE-xW-25-793000:b3n8vwWOKLSh-26-147000 create failed keys from task: [u'test_collections-1947', u'test_collections-1972', u'test_collections-1648'] 2021-05-03 04:29:07,635 | test | WARNING | Thread-5489 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:ialDE-xW-25-793000:0Db5KVwanwBiLYk-26-148000 create failed keys from task: [u'test_collections-2171', u'test_collections-1838', u'test_collections-2426', u'test_collections-1810', u'test_collections-2219', u'test_collections-1980', u'test_collections-2389'] 2021-05-03 04:29:07,648 | test | WARNING | Thread-5539 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Z5nFu5-0-245000:4W-yYT_z%WqtYH9q7tMt-0-249000 delete failed keys from task: [u'test_collections-1'] 2021-05-03 04:29:08,190 | test | WARNING | Thread-5567 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F-TqNZo-0-286000 delete failed keys from task: [u'test_collections-48', u'test_collections-338', u'test_collections-114', u'test_collections-111', u'test_collections-82', u'test_collections-154', u'test_collections-231', u'test_collections-182', u'test_collections-271'] 2021-05-03 04:29:08,267 | test | WARNING | Thread-5572 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:HXU0-YufiN-0-283000 update failed keys from task: [u'test_collections-706', u'test_collections-905', u'test_collections-836', u'test_collections-969', u'test_collections-876', u'test_collections-865', u'test_collections-820', u'test_collections-873'] 2021-05-03 04:29:08,526 | test | WARNING | Thread-5602 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:L4-27-823000:MHNGdBEFiso0bCd-28-17000 create failed keys from task: [u'test_collections-1048'] 2021-05-03 04:29:08,529 | test | WARNING | Thread-5604 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:L4-27-823000:Uh3ADBjTehP9u-27-986000 create failed keys from task: [u'test_collections-1048'] 2021-05-03 04:29:08,532 | test | WARNING | Thread-5610 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 delete failed keys from task: [u'test_collections-245', u'test_collections-267', u'test_collections-125', u'test_collections-222', u'test_collections-160', u'test_collections-1', u'test_collections-61', u'test_collections-338', u'test_collections-459', u'test_collections-256', u'test_collections-136', u'test_collections-378', u'test_collections-178', u'test_collections-173', u'test_collections-493'] 2021-05-03 04:29:08,532 | test | WARNING | Thread-5611 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:xXKqta-0-302000 update failed keys from task: [u'test_collections-849', u'test_collections-537', u'test_collections-515', u'test_collections-626', u'test_collections-524', u'test_collections-666', u'test_collections-589', u'test_collections-663', u'test_collections-883', u'test_collections-550'] 2021-05-03 04:29:08,540 | test | WARNING | Thread-5637 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:QoV_-0-305000 update failed keys from task: [u'test_collections-635', u'test_collections-503', u'test_collections-630', u'test_collections-675', u'test_collections-650'] 2021-05-03 04:29:08,542 | test | WARNING | Thread-5640 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 update failed keys from task: [u'test_collections-809', u'test_collections-849', u'test_collections-768', u'test_collections-601', u'test_collections-644', u'test_collections-886', u'test_collections-881', u'test_collections-561', u'test_collections-913', u'test_collections-779', u'test_collections-956', u'test_collections-579', u'test_collections-612', u'test_collections-931', u'test_collections-657', u'test_collections-974', u'test_collections-890'] 2021-05-03 04:29:07,628 | test | WARNING | Thread-5486 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:1VWkDzXAIVvBXF%X%l-25-311000:GvHrsBSZkvFIlv-25-430000 create failed keys from task: [u'test_collections-2850'] 2021-05-03 04:29:08,528 | test | WARNING | Thread-5603 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:L4-27-823000:cUHwFNZXvHl8pVuzY-29-163000 create failed keys from task: [u'test_collections-1048'] 2021-05-03 04:29:08,605 | test | WARNING | Thread-5565 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:kLJIUscF1_r-26-317000:vxaslEClNsG-26-511000 create failed keys from task: [u'test_collections-2132'] 2021-05-03 04:29:08,605 | test | WARNING | Thread-5566 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:kLJIUscF1_r-26-317000:UfE4L0Ez3Ka%G8C-26-510000 create failed keys from task: [u'test_collections-2850'] 2021-05-03 04:29:08,664 | test | WARNING | Thread-5572 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:HXU0-YufiN-0-283000 delete failed keys from task: [u'test_collections-426', u'test_collections-258', u'test_collections-468', u'test_collections-463', u'test_collections-471'] 2021-05-03 04:29:08,667 | test | WARNING | Thread-5637 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:QoV_-0-305000 delete failed keys from task: [u'test_collections-39', u'test_collections-177'] 2021-05-03 04:29:08,668 | test | WARNING | Thread-5610 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:XVVjWkmbH0e-0-305000 create failed keys from task: [u'test_collections-2767', u'test_collections-2584'] 2021-05-03 04:29:08,680 | test | WARNING | Thread-5611 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:xXKqta-0-302000 delete failed keys from task: [u'test_collections-87', u'test_collections-116', u'test_collections-113', u'test_collections-234', u'test_collections-114', u'test_collections-91', u'test_collections-166', u'test_collections-241', u'test_collections-271'] 2021-05-03 04:29:08,759 | test | WARNING | Thread-5640 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 delete failed keys from task: [u'test_collections-338', u'test_collections-101', u'test_collections-493', u'test_collections-160'] 2021-05-03 04:29:08,823 | test | WARNING | Thread-5640 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:3ADtwdSt74pdAWQ9R-0-301000:8UHHE8bg5EGGO-0-304000 create failed keys from task: [u'test_collections-2755'] 2021-05-03 04:29:22,661 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 04:30:39,604 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 04:30:39,605 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 04:31:09,355 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 04:31:09,357 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-03 04:31:35,375 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 04:31:35,375 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 04:32:36,059 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 04:32:55,184 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2060000896 seconds to complete 2021-05-03 04:32:55,203 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok During the test, Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 ---------------------------------------------------------------------- Ran 1 test in 2490.780s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_2 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_3', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 3, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 04:33:06,164 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 04:33:07,484 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 04:33:08,345 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb9776 type=client open_count=1 channel=[id: 0x512df523, 0.0.0.0/0.0.0.0:51119] timeout=300.0>) 2021-05-03 04:33:15,144 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb9777 type=client open_count=1 channel=[id: 0xad86bbe8, 0.0.0.0/0.0.0.0:45242] timeout=300.0>) 2021-05-03 04:33:17,209 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb9778 type=client open_count=1 channel=[id: 0x56fb0dc7, 0.0.0.0/0.0.0.0:50815] timeout=300.0>) 2021-05-03 04:33:19,230 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb9779 type=client open_count=1 channel=[id: 0xe7cfed25, 0.0.0.0/0.0.0.0:45837] timeout=300.0>) 2021-05-03 04:33:21,256 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb977a type=client open_count=1 channel=[id: 0x0ae5cd2d, 0.0.0.0/0.0.0.0:60291] timeout=300.0>) 2021-05-03 04:33:23,286 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb977b type=client open_count=1 channel=[id: 0x97637611, 0.0.0.0/0.0.0.0:44403] timeout=300.0>) 2021-05-03 04:33:26,319 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb977c type=client open_count=1 channel=[id: 0xd9b46842, 0.0.0.0/0.0.0.0:47759] timeout=300.0>) 2021-05-03 04:33:28,391 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb977d type=client open_count=1 channel=[id: 0xc51bba84, 0.0.0.0/0.0.0.0:41953] timeout=300.0>) 2021-05-03 04:33:30,426 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xb977e type=client open_count=1 channel=[id: 0xfc59d4fd, 0.0.0.0/0.0.0.0:58225] timeout=300.0>) 2021-05-03 04:33:32,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb977f type=client open_count=1 channel=[id: 0x116c2fe2, 0.0.0.0/0.0.0.0:54537] timeout=300.0>) 2021-05-03 04:33:41,996 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9780 type=client open_count=1 channel=[id: 0xed5ac408, 0.0.0.0/0.0.0.0:50243] timeout=300.0>) 2021-05-03 04:33:44,029 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9781 type=client open_count=1 channel=[id: 0x0ee737c1, 0.0.0.0/0.0.0.0:45419] timeout=300.0>) 2021-05-03 04:33:46,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9782 type=client open_count=1 channel=[id: 0xd1744882, 0.0.0.0/0.0.0.0:54077] timeout=300.0>) 2021-05-03 04:33:48,081 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9783 type=client open_count=1 channel=[id: 0x6b13a228, 0.0.0.0/0.0.0.0:43714] timeout=300.0>) 2021-05-03 04:33:50,132 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9784 type=client open_count=1 channel=[id: 0xf4ec5ce7, 0.0.0.0/0.0.0.0:60457] timeout=300.0>) 2021-05-03 04:33:52,171 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9785 type=client open_count=1 channel=[id: 0x776be199, 0.0.0.0/0.0.0.0:52421] timeout=300.0>) 2021-05-03 04:33:54,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9786 type=client open_count=1 channel=[id: 0x04da95d0, 0.0.0.0/0.0.0.0:32825] timeout=300.0>) 2021-05-03 04:33:56,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0xb9787 type=client open_count=1 channel=[id: 0xe00b5362, 0.0.0.0/0.0.0.0:39483] timeout=300.0>) 2021-05-03 04:33:58,305 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb9788 type=client open_count=1 channel=[id: 0xebeebb9e, 0.0.0.0/0.0.0.0:40435] timeout=300.0>) 2021-05-03 04:34:08,394 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb9789 type=client open_count=1 channel=[id: 0xcea4bc63, 0.0.0.0/0.0.0.0:43956] timeout=300.0>) 2021-05-03 04:34:10,424 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb978a type=client open_count=1 channel=[id: 0x2d13da83, 0.0.0.0/0.0.0.0:36840] timeout=300.0>) 2021-05-03 04:34:12,453 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb978b type=client open_count=1 channel=[id: 0xcf1e586e, 0.0.0.0/0.0.0.0:43147] timeout=300.0>) 2021-05-03 04:34:14,496 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb978c type=client open_count=1 channel=[id: 0xfe763e88, 0.0.0.0/0.0.0.0:54688] timeout=300.0>) 2021-05-03 04:34:16,596 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb978d type=client open_count=1 channel=[id: 0xd42edf5e, 0.0.0.0/0.0.0.0:57187] timeout=300.0>) 2021-05-03 04:34:18,622 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb978e type=client open_count=1 channel=[id: 0xe4120aaa, 0.0.0.0/0.0.0.0:54989] timeout=300.0>) 2021-05-03 04:34:20,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb978f type=client open_count=1 channel=[id: 0x846c1e0d, 0.0.0.0/0.0.0.0:56855] timeout=300.0>) 2021-05-03 04:34:22,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb9790 type=client open_count=1 channel=[id: 0x20a240af, 0.0.0.0/0.0.0.0:35389] timeout=300.0>) 2021-05-03 04:34:24,714 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0xb9791 type=client open_count=1 channel=[id: 0x26c0ff25, 0.0.0.0/0.0.0.0:60880] timeout=300.0>) 2021-05-03 04:34:26,766 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9792 type=client open_count=1 channel=[id: 0x457cdc84, 0.0.0.0/0.0.0.0:50977] timeout=300.0>) 2021-05-03 04:34:35,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9793 type=client open_count=1 channel=[id: 0xa9cd9afd, 0.0.0.0/0.0.0.0:52770] timeout=300.0>) 2021-05-03 04:34:37,219 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9794 type=client open_count=1 channel=[id: 0xec65962d, 0.0.0.0/0.0.0.0:43273] timeout=300.0>) 2021-05-03 04:34:39,246 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9795 type=client open_count=1 channel=[id: 0x99c57dec, 0.0.0.0/0.0.0.0:46375] timeout=300.0>) 2021-05-03 04:34:41,280 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9796 type=client open_count=1 channel=[id: 0xedc6c24e, 0.0.0.0/0.0.0.0:56109] timeout=300.0>) 2021-05-03 04:34:43,303 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9797 type=client open_count=1 channel=[id: 0xb93d98ff, 0.0.0.0/0.0.0.0:38375] timeout=300.0>) 2021-05-03 04:34:45,334 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9798 type=client open_count=1 channel=[id: 0xebbeaf45, 0.0.0.0/0.0.0.0:51729] timeout=300.0>) 2021-05-03 04:34:47,382 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb9799 type=client open_count=1 channel=[id: 0x68254756, 0.0.0.0/0.0.0.0:49057] timeout=300.0>) 2021-05-03 04:34:49,404 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb979a type=client open_count=1 channel=[id: 0x389683df, 0.0.0.0/0.0.0.0:48615] timeout=300.0>) 2021-05-03 04:34:51,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0xb979b type=client open_count=1 channel=[id: 0x3a0ae2f8, 0.0.0.0/0.0.0.0:47955] timeout=300.0>) 2021-05-03 04:34:53,448 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb979c type=client open_count=1 channel=[id: 0xb040d416, 0.0.0.0/0.0.0.0:52545] timeout=300.0>) 2021-05-03 04:34:59,267 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb979d type=client open_count=1 channel=[id: 0x0296e527, 0.0.0.0/0.0.0.0:47609] timeout=300.0>) 2021-05-03 04:35:01,292 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb979e type=client open_count=1 channel=[id: 0xd1df2f52, 0.0.0.0/0.0.0.0:58931] timeout=300.0>) 2021-05-03 04:35:03,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb979f type=client open_count=1 channel=[id: 0xcdba4ab8, 0.0.0.0/0.0.0.0:59799] timeout=300.0>) 2021-05-03 04:35:05,332 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb97a0 type=client open_count=1 channel=[id: 0xcc4d6524, 0.0.0.0/0.0.0.0:37975] timeout=300.0>) 2021-05-03 04:35:07,367 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0xb97a1 type=client open_count=1 channel=[id: 0xef0d17f0, 0.0.0.0/0.0.0.0:33677] timeout=300.0>) 2021-05-03 04:35:09,390 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 04:35:26,282 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 04:35:40,635 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 04:35:40,687 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 04:35:41,006 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 04:36:12,809 | test | INFO | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 12.4905944319 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 04:36:12,835 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 04:36:17,890 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 04:36:27,979 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 04:36:28,022 | test | INFO | pool-5-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.2130000591 sec 2021-05-03 04:36:28,362 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 04:37:10,312 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 04:37:10,599 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 592512432 | 80169283 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 565022672 | 102899182 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 641224048 | 56965678 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-03 04:37:10,601 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 04:37:57,821 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 04:37:57,836 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 04:53:30,257 | test | INFO | pool-5-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *.......*................*......*X | 16.195 | | 12.400 | .....................X | 12.400 | | 3.600 | .....*X | 3.600 | | 0.300 | *X | 0.300 | | 0.000 | *************************************************..***X | 1.500 | | 1.100 | X | 1.100 | | 0.300 | *X | 0.300 | | 0.000 | ***************.*...***X | 2.700 | | 1.901 | ..X | 1.901 | | 0.300 | ***.....*..........**...*X | 10.201 | | 7.801 | .............X | 7.801 | | 3.000 | ....*X | 3.000 | | 1.500 | .**..****X | 2.600 | | 1.000 | **..*.................*.................................................**.................**.....X | 50.596 | | 39.796 | .......................................................................*X | 39.796 | | 33.177 | ...........................................................*X | 33.177 | | 20.781 | ....................................X | 20.781 | | 4.181 | ......X | 4.181 | | 0.000 | *****************************************************************************************************************************************.**.X | 1.500 | | 1.200 | .**...*.....*..........**.....*..........*X | 19.500 | | 14.400 | .........................*X | 14.400 | | 0.000 | ************************************..***X | 1.500 | | 1.100 | X | 1.100 | | 0.300 | X | 0.300 | | 0.000 | ****************.*.*X | 1.500 | | 1.200 | .**.*......**...X | 6.697 | | 6.297 | ..........X | 6.297 | | 2.300 | ...*X | 2.300 | | 1.400 | .**......*..............**....*X | 14.389 | | 10.389 | .................X | 10.389 | | 2.400 | ...*X | 2.400 | | 0.000 | *......*................**......*X | 15.703 | | 12.901 | ......................X | 12.901 | | 8.600 | ..............**...........**........................*...................................X | 47.183 | | 45.381 | .................................................................................**....*..............*X | 55.129 | | 53.329 | ...............................................................................................*X | 53.329 | | 37.311 | ..................................................................X | 37.311 | | 5.700 | .........*X | 5.700 | | 0.000 | **********************************************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 04:53:32,099 | test | INFO | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...........*..................**............................*........................................**...*X | 1237.423 | | 1125.200 | .........................................................................................X | 1125.200 | | 918.572 | .........................................................................X | 918.572 | | 898.272 | .......................................................................*X | 898.272 | | 888.172 | ......................................................................*....*X | 935.843 | | 927.143 | .........................................................................*X | 927.143 | | 911.943 | ........................................................................X | 911.943 | | 865.022 | ....................................................................X | 865.022 | | 841.022 | ..................................................................*X | 841.022 | | 825.722 | .................................................................X | 825.722 | | 778.049 | .............................................................**.*X | 784.109 | | 768.409 | .............................................................*.X | 789.359 | | 779.939 | ..............................................................*X | 779.939 | | 774.239 | .............................................................X | 774.239 | | 687.621 | ......................................................*X | 687.621 | | 628.561 | .................................................*X | 628.561 | | 555.661 | ...........................................X | 555.661 | | 457.749 | ...................................X | 457.749 | | 420.858 | .................................*X | 420.858 | | 415.758 | ................................X | 415.758 | | 401.516 | ...............................**..**.......****..*....**.*X | 602.240 | | 593.200 | ..............................................*.*.**X | 609.085 | | 568.648 | ............................................*X | 568.648 | | 564.706 | ............................................*X | 564.706 | | 560.706 | ............................................**.*X | 578.245 | | 570.845 | .............................................X | 570.845 | | 565.446 | ............................................*X | 565.446 | | 546.046 | ...........................................**..*...*.*X | 622.524 | | 570.472 | .............................................X | 570.472 | | 461.096 | ....................................X | 461.096 | | 425.996 | .................................*X | 425.996 | | 421.852 | .................................X | 421.852 | | 413.257 | ................................**.*X | 423.157 | | 419.241 | ................................*....*X | 461.882 | | 457.482 | ...................................*.X | 462.840 | | 433.557 | ..................................**.**..........***.X | 589.545 | | 567.745 | ............................................X | 567.745 | | 531.545 | .........................................*X | 531.545 | | 522.952 | .........................................**.*.....**...*X | 640.900 | | 630.900 | .................................................X | 630.900 | | 569.000 | ............................................X | 569.000 | | 496.400 | .......................................*X | 496.400 | | 459.168 | ....................................X | 459.168 | | 426.368 | .................................**..**.......***..X | 565.359 | | 547.817 | ...........................................X | 547.817 | | 502.369 | .......................................*X | 502.369 | | 482.523 | .....................................*X | 482.523 | | 469.723 | ....................................X | 469.723 | | 465.800 | ....................................*X | 465.800 | | 446.100 | ...................................*.*.**..**.**.**....*....X | 628.269 | | 622.769 | .................................................*X | 622.769 | | 610.982 | ................................................*X | 610.982 | | 592.651 | ..............................................*..*X | 610.651 | | 607.700 | ................................................**X | 607.700 | | 570.778 | .............................................X | 570.778 | | 520.735 | .........................................X | 520.735 | | 453.435 | ...................................****.**X | 465.637 | | 438.313 | ..................................X | 438.313 | | 424.065 | .................................***.**.......***...*...*.....**..*X | 692.900 | | 680.800 | ......................................................X | 680.800 | | 643.200 | ..................................................X | 643.200 | | 628.100 | .................................................**X | 628.100 | | 626.800 | .................................................*.X | 631.200 | | 583.700 | ..............................................*X | 583.700 | | 531.668 | .........................................X | 531.668 | | 411.703 | ................................*X | 411.703 | | 394.503 | ..............................**....**.*X | 448.705 | | 441.041 | ..................................*****...*.**..**...*.....*.*X | 626.326 | | 605.485 | ...............................................X | 605.485 | | 555.245 | ...........................................X | 555.245 | | 532.545 | ..........................................**X | 532.545 | | 455.233 | ...................................*X | 455.233 | | 430.433 | .................................*X | 430.433 | | 405.633 | ...............................X | 405.633 | | 385.400 | ..............................X | 385.400 | | 377.900 | .............................**.*..**.*X | 422.600 | | 417.000 | ................................*.**..*..*....****.*X | 544.153 | | 543.923 | ..........................................**.*.*..**.**.X | 611.400 | | 582.916 | ..............................................*X | 582.916 | | 525.316 | .........................................X | 525.316 | | 433.116 | ..................................X | 433.116 | | 399.337 | ...............................**.****X | 416.700 | | 404.300 | ...............................*..*..**.....*.....**.*X | 586.338 | | 535.312 | ..........................................****..*..X | 582.586 | | 580.286 | .............................................*X | 580.286 | | 560.186 | ............................................*X | 560.186 | | 526.055 | .........................................X | 526.055 | | 491.555 | ......................................*X | 491.555 | | 471.555 | .....................................X | 471.555 | | 425.500 | .................................*X | 425.500 | | 421.200 | .................................X | 421.200 | | 411.081 | ................................X | 411.081 | | 408.881 | ................................*X | 408.881 | | 407.581 | ...............................**..*...*.**..*...**...**..*...*.*X | 648.400 | | 638.500 | ..................................................X | 638.500 | | 619.300 | .................................................*X | 619.300 | | 576.800 | .............................................X | 576.800 | | 574.500 | .............................................X | 574.500 | | 539.400 | ..........................................***X | 540.500 | | 467.600 | ....................................X | 467.600 | | 382.500 | .............................X | 382.500 | | 349.900 | ...........................**...*..**..***...*X | 475.800 | | 466.200 | ....................................X | 466.200 | | 458.600 | ....................................X | 458.600 | | 415.300 | ................................**.**.*.....****..**..**.*.*X | 570.116 | | 554.716 | ...........................................*X | 554.716 | | 492.816 | ......................................X | 492.816 | | 385.770 | ..............................X | 385.770 | | 328.370 | .........................**..*..**..*X | 405.840 | | 386.357 | ..............................X | 386.357 | | 379.291 | .............................*.**.*.*X | 417.301 | | 394.575 | ..............................*X | 394.575 | | 394.357 | ..............................*....**....*..X | 514.103 | | 491.616 | ......................................*X | 491.616 | | 477.716 | .....................................**.*......***X | 568.172 | | 563.672 | ............................................X | 563.672 | | 501.238 | .......................................X | 501.238 | | 481.387 | .....................................*X | 481.387 | | 442.187 | ..................................X | 442.187 | | 404.854 | ...............................*X | 404.854 | | 386.932 | ..............................**.X | 406.032 | | 405.877 | ...............................**.X | 415.777 | | 407.477 | ...............................*.**.**.....**...**.......*.....X | 670.323 | | 665.623 | ....................................................*X | 665.623 | | 619.023 | .................................................X | 619.023 | | 587.300 | ..............................................**..**...*.....X | 715.200 | | 695.700 | .......................................................*X | 695.700 | | 649.300 | ...................................................X | 649.300 | | 596.239 | ...............................................****...*.*X | 644.765 | | 634.765 | ..................................................*X | 634.765 | +----------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 04:53:32,154 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *....*........................................................*.....................**...................X | 65.918 | | 29.118 | ...........................................*X | 29.118 | | 15.318 | ......................*X | 15.318 | | 0.000 | ******************************************************************************************************...................**..........................*....................................................X | 64.495 | | 51.595 | .............................................................................**X | 51.595 | | 34.400 | ...................................................X | 34.400 | | 0.000 | ********************************************************************************************************************************************************************************************.**.*X | 1.500 | | 1.100 | X | 1.100 | | 0.300 | *X | 0.300 | | 0.000 | ***********....*.......................**................**..............X | 37.600 | | 36.000 | .....................................................*X | 36.000 | | 25.200 | .....................................X | 25.200 | | 13.600 | ...................X | 13.600 | | 2.398 | ..**..**.X | 4.498 | | 2.100 | ..****..*.......X | 8.394 | | 7.794 | ..........*X | 7.794 | | 5.394 | .......X | 5.394 | | 0.600 | *X | 0.600 | | 0.000 | **.*...**..X | 4.101 | | 3.700 | ....X | 3.700 | | 2.900 | ...*X | 2.900 | | 2.600 | ..**.X | 2.999 | | 1.399 | .*X | 1.399 | | 0.799 | **...*.......................**..........*...X | 26.806 | | 13.199 | ...................*X | 13.199 | | 7.799 | ..........*X | 7.799 | | 3.399 | ....X | 3.399 | | 1.500 | .*X | 1.500 | | 1.200 | **..*...............**...........*...............*.......*X | 33.821 | | 28.121 | .........................................*X | 28.121 | | 19.321 | ............................X | 19.321 | | 9.000 | ............**X | 9.000 | | 6.600 | .........X | 6.600 | | 1.800 | .*X | 1.800 | | 0.000 | ************************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 04:53:34,335 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 44.3689574079 | 3693387776 | 2479984640 | 0 / 3909087232 | 126800 / 256390 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 45.3752642706 | 3693387776 | 2548391936 | 0 / 3909087232 | 125750 / 256071 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 35.0407747197 | 6078930944 | 4793126912 | 0 / 3758092288 | 126800 / 248200 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 39.8248325605 | 4201836544 | 3107188736 | 0 / 3758092288 | 126400 / 250800 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 04:53:50,948 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 04:55:51,427 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155672768 | 253865197 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155359680 | 260723342 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 432512240 | 451936452 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-03 04:55:51,430 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-05-03 04:56:01,615 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during swap_rebalance 2021-05-03 04:56:01,615 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : swap_rebalance 2021-05-03 04:56:09,918 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 04:56:10,338 | test | INFO | pool-5-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 15.1921132457 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 14.6751592357 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 15.0063051702 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 0.777331995988 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 16.7389640214 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-03 04:56:10,424 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 04:56:15,674 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 04:56:21,190 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 04:56:21,813 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.25 2021-05-03 04:56:26,980 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 3.13 2021-05-03 04:56:32,207 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 5.51 2021-05-03 04:56:37,400 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 7.81 2021-05-03 04:56:40,401 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:UqqoKzC4wdVyYlJRUOv-28-460000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:42,512 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 10.32 2021-05-03 04:56:45,345 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:UqqoKzC4wdVyYlJRUOv-28-460000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:45,512 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +--------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:45,513 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:45,526 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +--------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +--------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:45,528 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:47,678 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 12.93 2021-05-03 04:56:50,467 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-206 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-396 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:50,469 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-206 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-396 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:50,480 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-206 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-396 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:50,480 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-206 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-396 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:51,522 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-540 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:51,523 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-540 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:51,522 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-540 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:51,529 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-540 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:52,979 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 15.98 2021-05-03 04:56:55,872 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:55,872 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:55,874 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:55,875 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:57,714 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:57,716 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:57,732 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:57,733 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:58,262 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 19.06 2021-05-03 04:56:59,852 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1050 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:59,854 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1050 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:59,861 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1050 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:56:59,861 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1050 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:02,279 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:02,279 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:02,279 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:02,289 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:03,502 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 22.34 2021-05-03 04:57:04,141 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:04,142 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:04,144 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1523 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:04,142 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1523 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:05,881 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1608 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1798 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1704 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1694 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:05,881 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1608 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1798 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1704 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1694 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:05,887 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1608 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1798 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1704 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1694 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:05,888 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1608 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1798 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1704 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1694 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:07,321 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1806 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1996 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1821 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:07,322 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1806 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1996 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1821 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:07,381 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1806 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1996 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1821 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:07,381 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1806 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1996 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1821 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:08,742 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 25.72 2021-05-03 04:57:12,085 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:12,085 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2491 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2501 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:13,285 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:13,286 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:13,303 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:13,305 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:13,905 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 28.58 2021-05-03 04:57:16,437 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:16,438 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3053 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:16,447 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:16,448 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3053 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:19,369 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 31.29 2021-05-03 04:57:22,390 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:22,391 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3628 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:22,405 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:22,405 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3628 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:24,641 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 04:57:29,960 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 04:57:37,792 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 34.15 2021-05-03 04:57:43,125 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 37.82 2021-05-03 04:57:43,523 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:43,523 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3465 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:48,378 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 40.5 2021-05-03 04:57:51,434 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3929 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3972 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,436 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3929 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3972 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,443 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3929 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3972 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,444 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3929 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3972 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,450 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3988 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3945 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,450 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3818 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3988 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3945 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,450 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3988 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3945 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,453 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3818 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3988 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3945 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,490 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3929 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3972 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:51,493 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3929 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3972 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:52,430 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:11pz-17-672000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3611 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3781 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:52,430 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:11pz-17-672000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3611 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3781 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:52,877 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ss1t5-zIAB05rSYnjPP-28-441000:m7eHMb82CPP-28-442000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:52,904 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in bucket2:_default:E1EtV9VVfZlKs0KEAF3w-28-427000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:52,927 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in bucket2:_default:CqbWtD0tt5BgtQN5-28-426000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:53,644 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 43.17 2021-05-03 04:57:54,914 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,915 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4012 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,920 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,921 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4012 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,937 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,937 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4012 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,944 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,944 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4012 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,951 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:54,953 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4012 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,262 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,263 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,272 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,273 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,282 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,282 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,283 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,285 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4234 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:57:58,937 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 46.16 2021-05-03 04:58:01,280 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4410 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:01,282 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4473 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4410 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,174 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4410 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,174 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4473 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4410 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,181 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4410 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,183 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4473 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4410 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,369 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4410 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,371 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4473 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4410 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,385 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4410 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,388 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4473 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4410 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,523 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:11pz-17-672000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:02,525 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:11pz-17-672000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4291 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4301 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:04,151 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 48.88 2021-05-03 04:58:04,970 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:04,971 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4757 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:04,980 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:04,982 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4757 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:05,010 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:05,012 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:05,015 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:05,015 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:09,240 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4849 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:09,242 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4849 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:09,525 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 51.52 2021-05-03 04:58:10,354 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:11pz-17-672000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:10,355 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in bucket2:-%-17-541000:11pz-17-672000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4724 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:15,769 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 54.57 2021-05-03 04:58:20,957 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 56.86 2021-05-03 04:58:26,108 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 59.83 2021-05-03 04:58:31,351 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 63.11 2021-05-03 04:58:36,510 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.83 2021-05-03 04:58:41,694 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 04:58:46,868 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 04:58:53,642 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 67.48 2021-05-03 04:58:54,563 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope1:p91eF4no-28-469000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-496 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:54,701 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-526 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:57,618 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:57,618 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1204 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1394 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:58,710 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-125 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:58,848 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.69 2021-05-03 04:58:59,032 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-637 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-655 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,276 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope2:7ygvlc1Li30FxiV-28-472000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-247 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-369 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,411 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,487 | test | INFO | pool-5-thread-1 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,489 | test | INFO | pool-5-thread-1 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2609 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2536 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,641 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,776 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-38 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-127 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-100 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-93 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-82 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-142 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,838 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2629 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,839 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2629 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:58:59,954 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-535 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-689 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-525 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-655 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-520 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-672 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:00,444 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:scope2:7ygvlc1Li30FxiV-28-472000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:01,065 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-459 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:01,065 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-459 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-478 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-560 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:01,953 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope2:x%ashpqwLrSRhJIqm-28-473000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-603 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-615 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-650 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:01,986 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:12-OWL-28-477000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-603 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-615 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-650 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-561 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:03,322 | test | INFO | pool-5-thread-25 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-378 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:03,384 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:03,990 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope2:7ygvlc1Li30FxiV-28-472000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-439 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-448 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,052 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 73.39 2021-05-03 04:59:04,221 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-828 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-879 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-846 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-790 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,286 | test | INFO | pool-5-thread-1 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2730 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,288 | test | INFO | pool-5-thread-1 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2730 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,325 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-924 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-976 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,331 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:scope2:7ygvlc1Li30FxiV-28-472000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-976 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,341 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-256 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,447 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2853 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2760 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,450 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2853 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2731 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2760 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:04,790 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:yV5HcmKzB3if3-28-476000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-645 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-643 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-600 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-653 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:05,303 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-934 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:05,407 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-698 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:05,408 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-698 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:07,030 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope2:x%ashpqwLrSRhJIqm-28-473000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-803 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-784 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-796 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:07,286 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:Z-28-474000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-533 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-643 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-644 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-564 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-530 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:07,857 | test | INFO | pool-5-thread-16 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-487 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:07,974 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-28 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-108 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-136 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-140 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-173 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:08,040 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-923 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-965 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:08,042 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:_default:F8P_F5H8QOZ-28-467000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-923 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-931 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-965 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:09,263 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 76.05 2021-05-03 04:59:09,612 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-136 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-140 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-173 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:09,680 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:09,680 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1656 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:10,733 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:yV5HcmKzB3if3-28-476000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-759 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-748 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-897 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-884 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-791 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-792 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:10,923 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-601 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-632 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-644 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-552 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:11,375 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:k3%-19-245000:DePVvM-19-548000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3636 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:11,377 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:k3%-19-245000:DePVvM-19-548000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3636 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:11,815 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope2:x%ashpqwLrSRhJIqm-28-473000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-965 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:12,131 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-878 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:12,132 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-878 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:12,387 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:scope2:pqQlBLVhIRfhPnlG-28-473000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-677 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-575 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-652 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:13,190 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-349 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-248 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-276 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-341 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:14,351 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-248 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-276 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-341 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:14,461 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 78.13 2021-05-03 04:59:14,667 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:yV5HcmKzB3if3-28-476000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-951 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:14,971 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-728 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-859 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-705 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-740 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:15,316 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:scope1:E_InIFQkSOujfh-IAS-28-469000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-640 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-695 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:15,336 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:k3%-19-245000:DePVvM-19-548000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3913 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3883 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:15,338 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:k3%-19-245000:DePVvM-19-548000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3913 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3936 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3883 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:17,168 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:Z-28-474000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-839 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-705 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-740 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:17,536 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:12-OWL-28-477000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-740 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-892 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:17,713 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4190 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4000 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:17,713 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4190 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4000 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:17,743 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:vq-28-469000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-529 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-537 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-512 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:18,467 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:scope1:vq-28-469000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-59 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-62 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-199 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-154 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:19,210 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-408 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:19,352 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:pz0mDCN1wF0J0XZFtn-28-467000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-48 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-62 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-165 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:19,625 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:19,625 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2095 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2105 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:19,658 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 80.37 2021-05-03 04:59:20,213 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-408 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:20,420 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:scope1:E_InIFQkSOujfh-IAS-28-469000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-739 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-726 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-713 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-756 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:20,551 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-940 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:22,509 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:12-OWL-28-477000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-917 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:22,628 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:A_-19-946000:FFb5-20-102000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:22,630 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:A_-19-946000:FFb5-20-102000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1132 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:22,650 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:k3%-19-245000:DePVvM-19-548000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4261 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:22,650 | test | INFO | pool-5-thread-14 | [table_view:display:72] Keys failed in default:k3%-19-245000:DePVvM-19-548000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4261 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:22,796 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:vq-28-469000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-847 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-768 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-835 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-887 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-756 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-895 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:22,816 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:Z-28-474000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-964 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:23,197 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:_default:pz0mDCN1wF0J0XZFtn-28-467000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-357 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-312 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-282 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:23,207 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:scope1:vq-28-469000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-357 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-312 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-282 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:23,302 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope2:9VB-28-472000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-676 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-690 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:23,308 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:scope2:pqQlBLVhIRfhPnlG-28-473000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-835 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-799 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-700 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-881 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-870 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:24,767 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 81.91 2021-05-03 04:59:24,990 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:scope1:vq-28-469000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:26,102 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2222 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:26,104 | test | INFO | pool-5-thread-29 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:W-J0er-20-490000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2222 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:26,121 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:OJ0q313-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-666 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-542 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:27,016 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:scope1:E_InIFQkSOujfh-IAS-28-469000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-952 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:27,750 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:OJ0q313-28-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-114 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:27,772 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:_default:v-28-465000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-54 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-114 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:27,871 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:_default:v-28-465000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-666 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:29,993 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 84.49 2021-05-03 04:59:30,102 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope2:9VB-28-472000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-848 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-855 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:30,473 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:OJ0q313-28-475000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-848 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-855 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:30,493 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1518 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1488 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:30,494 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1518 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1488 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:32,088 | test | INFO | pool-5-thread-10 | [table_view:display:72] Keys failed in default:_default:v-28-465000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-836 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:32,298 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:scope2:collection1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2546 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:32,299 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:scope2:collection1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2546 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:32,559 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:OJ0q313-28-475000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-340 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-231 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-240 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:32,575 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:_default:v-28-465000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-340 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-231 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-240 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:34,967 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:6ww7zVgOfHMKZhF-28-468000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-627 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:35,210 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 86.87 2021-05-03 04:59:35,450 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:_default:1Guk0rC8jODFV-28-467000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:35,575 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4411 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:35,576 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4411 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:36,561 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1647 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:36,562 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:37,200 | test | INFO | pool-5-thread-1 | [table_view:display:72] Keys failed in default:scope2:collection1 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-372 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:38,390 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:OJ0q313-28-475000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:38,391 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:_default:v-28-465000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:40,493 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 89.42 2021-05-03 04:59:40,914 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-03 04:59:41,802 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:41,803 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4790 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4600 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:42,023 | test | INFO | pool-5-thread-18 | [table_view:display:72] Keys failed in default:scope2:9VB-28-472000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:42,135 | test | INFO | pool-5-thread-7 | [table_view:display:72] Keys failed in default:_default:6ww7zVgOfHMKZhF-28-468000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-742 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:42,999 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1965 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:43,000 | test | INFO | pool-5-thread-3 | [table_view:display:72] Keys failed in default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1804 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1994 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1965 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:43,092 | test | INFO | pool-5-thread-22 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:UGBkg-28-477000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-634 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-671 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:43,246 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:UGBkg-28-477000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-88 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:43,749 | test | INFO | pool-5-thread-1 | [table_view:display:72] Keys failed in default:scope2:collection1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-435 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:44,931 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:lEbef7-28-476000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-189 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-184 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:45,704 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 91.82 2021-05-03 04:59:46,191 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope2:GiB-28-472000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2872 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:46,193 | test | INFO | pool-5-thread-20 | [table_view:display:72] Keys failed in default:scope2:GiB-28-472000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2872 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:47,079 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope2:GiB-28-472000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-789 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:48,885 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:A_-19-946000:FFb5-20-102000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1658 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:48,887 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:A_-19-946000:FFb5-20-102000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1658 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:49,229 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:49,230 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:k3%-19-245000:abs-19-641000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4944 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:49,378 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-03 04:59:49,380 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 10 docs from 0 to 5000 2021-05-03 04:59:49,381 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 12 docs from 0 to 5000 2021-05-03 04:59:49,381 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 19 docs from 0 to 5000 2021-05-03 04:59:49,382 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 21 docs from 0 to 5000 2021-05-03 04:59:49,384 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 6 docs from 0 to 5000 2021-05-03 04:59:49,384 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 12 docs from 0 to 5000 2021-05-03 04:59:49,385 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 11 docs from 0 to 5000 2021-05-03 04:59:49,387 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 3 to 5 2021-05-03 04:59:49,387 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 0 to 3 2021-05-03 04:59:49,388 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 12 docs from 0 to 5000 2021-05-03 04:59:49,401 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 3 to 5 2021-05-03 04:59:49,403 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 3 to 5 2021-05-03 04:59:49,404 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 3 to 5 2021-05-03 04:59:50,410 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:nF3ARDmo-EVo0-28-474000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:50,411 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:nF3ARDmo-EVo0-28-474000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2655 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:50,805 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 94.33 2021-05-03 04:59:50,825 | test | INFO | pool-5-thread-4 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:UGBkg-28-477000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-238 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-321 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:51,368 | test | INFO | pool-5-thread-1 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-682 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:53,025 | test | INFO | pool-5-thread-28 | [table_view:display:72] Keys failed in default:V9IuEbS-28-474000:lEbef7-28-476000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-209 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-399 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:53,766 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:scope2:GiB-28-472000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-495 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:54,342 | test | INFO | pool-5-thread-13 | [table_view:display:72] Keys failed in default:scope2:GiB-28-472000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-915 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:54,732 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:_default:yPhauax-28-466000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:54,733 | test | INFO | pool-5-thread-2 | [table_view:display:72] Keys failed in default:_default:yPhauax-28-466000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2551 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:55,989 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 97.1 2021-05-03 04:59:57,775 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:scope1:collection_1 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:57,776 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:scope1:collection_1 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:57,788 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:C-28-470000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:57,789 | test | INFO | pool-5-thread-12 | [table_view:display:72] Keys failed in default:scope1:C-28-470000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2502 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:59,700 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:A_-19-946000:FFb5-20-102000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1800 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1990 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 04:59:59,701 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in default:A_-19-946000:FFb5-20-102000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1800 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1990 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:00:01,119 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:00:01,907 | test | INFO | pool-5-thread-8 | [table_view:display:72] Keys failed in default:_default:K-KQOucmbVuZz-28-465000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-711 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-850 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:00:02,535 | test | INFO | pool-5-thread-5 | [table_view:display:72] Keys failed in default:scope1:eyc8g0SRUfoC3%B-28-469000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-578 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:00:02,615 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:scope1:C-28-470000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-714 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-747 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:00:07,292 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:00:13,648 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:00:23,875 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x11647f type=client open_count=1 channel=[id: 0x7ff8f0a6, 0.0.0.0/0.0.0.0:40489] timeout=300.0>) 2021-05-03 05:00:24,062 | test | ERROR | pool-5-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 05:00:26,427 | test | INFO | pool-5-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 256.090000153 sec 2021-05-03 05:00:30,444 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 7 docs from 0 to 5000 2021-05-03 05:00:54,135 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 13 docs from 0 to 5000 2021-05-03 05:00:54,142 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 05:00:54,144 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 4 docs from 0 to 5000 2021-05-03 05:00:54,145 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 05:00:54,154 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 05:00:54,155 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-03 05:00:54,155 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 05:00:54,157 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 05:00:54,157 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 20 docs from 500 to 980 2021-05-03 05:00:54,158 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-03 05:00:54,160 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 11 docs from 0 to 500 2021-05-03 05:00:54,161 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 05:00:54,161 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 4 docs from 2500 to 2900 2021-05-03 05:00:54,164 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 05:00:54,167 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 10 docs from 500 to 980 2021-05-03 05:00:54,167 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 14 docs from 500 to 980 2021-05-03 05:00:54,171 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 05:00:54,216 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 16 docs from 0 to 500 2021-05-03 05:00:54,219 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 9 docs from 500 to 980 2021-05-03 05:00:54,220 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 13 docs from 500 to 980 2021-05-03 05:00:54,220 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-03 05:00:54,223 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 9 docs from 500 to 980 2021-05-03 05:00:54,223 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 10 docs from 0 to 500 2021-05-03 05:00:54,226 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 12 docs from 500 to 980 2021-05-03 05:00:54,259 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 9 docs from 0 to 500 2021-05-03 05:00:54,262 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 5 docs from 500 to 980 2021-05-03 05:00:54,263 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 8 docs from 500 to 980 2021-05-03 05:00:54,265 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 6 docs from 500 to 980 2021-05-03 05:00:54,275 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-03 05:00:54,275 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 05:00:54,276 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-03 05:00:54,276 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 05:00:54,279 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 0 to 500 2021-05-03 05:00:54,279 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:00:54,279 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:00:54,280 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:00:54,282 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 05:00:54,283 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 05:00:54,283 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:00:54,285 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 05:00:54,285 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 7 docs from 0 to 500 2021-05-03 05:00:54,286 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 05:00:54,299 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 05:00:54,301 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 2 docs from 500 to 980 2021-05-03 05:00:54,309 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:00:54,309 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 05:00:54,309 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:00:54,311 | test | ERROR | pool-5-thread-21 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 05:01:39,256 | test | INFO | pool-5-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..**..*X | 23.298 | | 17.800 | ..*X | 19.498 | | 10.698 | .**.*.....*.......*.....*..........*...**...*.*.....**......**.**...........*........................................*X | 528.900 | | 477.606 | .........................................................................................X | 477.606 | | 383.206 | .......................................................................X | 383.206 | | 219.606 | ........................................X | 219.606 | | 180.358 | .................................X | 180.358 | | 169.058 | ..............................*......*X | 198.724 | | 185.666 | ..................................X | 185.666 | | 172.566 | ...............................X | 172.566 | | 168.132 | ..............................**.X | 169.532 | | 165.032 | ..............................X | 165.032 | | 148.500 | ...........................**.*X | 158.100 | | 155.469 | ............................*X | 155.469 | | 131.143 | .......................*X | 131.143 | | 129.087 | .......................**.*.......*.............**..................*...........X | 393.757 | | 370.711 | .....................................................................*X | 370.711 | | 269.729 | .................................................X | 269.729 | | 181.629 | .................................X | 181.629 | | 142.829 | ..........................*X | 142.829 | | 139.991 | .........................X | 139.991 | | 139.891 | .........................*X | 139.891 | | 128.474 | .......................**...*..**.*X | 163.205 | | 156.084 | ............................*X | 156.084 | | 128.600 | .......................**.*...*....*X | 169.970 | | 165.170 | ..............................X | 165.170 | | 140.370 | .........................X | 140.370 | | 113.917 | ....................**..**......*X | 154.200 | | 114.300 | ....................*X | 114.300 | | 59.900 | ..........X | 59.900 | | 23.700 | ...*X | 23.700 | | 9.900 | X | 9.900 | | 9.899 | X | 9.899 | | 9.299 | *X | 9.299 | | 8.899 | **.***X | 15.403 | | 12.800 | .X | 12.800 | | 9.000 | *X | 9.000 | | 6.700 | X | 6.700 | | 4.899 | **.***.**X | 13.403 | | 12.500 | .X | 12.500 | | 9.500 | *X | 9.500 | | 8.200 | *.***X | 14.898 | +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:01:41,163 | test | INFO | pool-5-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....**..................*X | 510.635 | | 472.035 | ...................X | 472.035 | | 353.983 | ..............X | 353.983 | | 219.268 | ........**..*...X | 338.241 | | 313.146 | ............X | 313.146 | | 247.546 | .........*X | 247.546 | | 150.996 | .....**....*.....*............X | 625.127 | | 594.127 | ........................*X | 594.127 | | 436.778 | .................****........*......*X | 744.281 | | 731.140 | ..............................X | 731.140 | | 722.803 | ..............................X | 722.803 | | 641.603 | ..........................X | 641.603 | | 579.931 | ........................X | 579.931 | | 414.786 | ................**.*...*.......X | 666.120 | | 604.950 | .........................X | 604.950 | | 580.850 | ........................X | 580.850 | | 487.050 | ....................*X | 487.050 | | 483.800 | ...................X | 483.800 | | 480.000 | ...................*...*...**.X | 641.743 | | 597.143 | ........................X | 597.143 | | 528.003 | .....................*X | 528.003 | | 485.155 | ....................**...**.....*...X | 741.536 | | 713.167 | .............................*X | 713.167 | | 700.577 | .............................**....*....X | 898.653 | | 824.553 | ..................................**.X | 832.657 | | 760.989 | ................................*.X | 785.089 | | 773.676 | ................................*.*.*X | 829.884 | | 805.693 | .................................X | 805.693 | | 586.300 | ........................*X | 586.300 | | 533.300 | ......................**X | 544.690 | | 544.690 | ......................X | 544.690 | | 531.590 | ......................X | 531.590 | | 524.490 | .....................X | 524.490 | | 380.957 | ...............**..*..........*..............**........X | 1154.400 | | 1113.000 | ...............................................X | 1113.000 | | 1025.100 | ...........................................**..**.........*..........*.......**.......*......X | 1982.151 | | 1982.151 | ....................................................................................*........**.*.....*X | 2300.100 | | 2292.300 | ..................................................................................................*.X | 2305.530 | | 2208.030 | ..............................................................................................*X | 2208.030 | | 2125.030 | ...........................................................................................X | 2125.030 | | 2094.755 | .........................................................................................X | 2094.755 | | 2060.155 | ........................................................................................**...X | 2141.655 | | 1998.400 | .....................................................................................X | 1998.400 | | 1899.000 | .................................................................................X | 1899.000 | | 1705.700 | ........................................................................**......*......**...*X | 2048.263 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 05:01:42,503 | test | INFO | pool-5-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....**....X | 46.199 | | 44.700 | .......*.**.*............*...........***......**.......**.......**.....*......*.......***......*............*...........X | 513.587 | | 484.205 | .............................................................................................X | 484.205 | | 407.045 | ..............................................................................X | 407.045 | | 351.045 | ...................................................................*X | 351.045 | | 351.009 | ...................................................................*..**.X | 366.438 | | 365.956 | ......................................................................X | 365.956 | | 360.056 | .....................................................................*....X | 385.157 | | 370.857 | .......................................................................X | 370.857 | | 306.257 | ..........................................................X | 306.257 | | 243.500 | ..............................................*X | 243.500 | | 68.600 | ............*X | 68.600 | | 20.898 | ...*X | 20.898 | | 3.698 | *X | 3.698 | | 1.200 | X | 1.200 | | 1.000 | ****X | 4.501 | | 2.701 | X | 2.701 | | 1.000 | **X | 3.201 | | 3.201 | *X | 4.701 | | 4.100 | X | 4.100 | | 3.500 | *X | 3.500 | | 2.999 | ****.*X | 5.504 | | 5.103 | *X | 5.103 | | 3.700 | X | 3.700 | | 2.700 | X | 2.700 | | 2.500 | *X | 2.500 | | 1.900 | X | 1.900 | | 1.200 | X | 1.200 | | 0.200 | ******.********.***.*X | 15.897 | | 14.702 | .X | 14.702 | | 14.502 | .X | 14.502 | | 12.502 | .*X | 12.502 | | 11.300 | .X | 11.300 | | 9.100 | *X | 9.100 | | 8.700 | *.****.**X | 18.304 | | 16.604 | ..X | 16.604 | | 9.700 | X | 9.700 | | 6.100 | X | 6.100 | | 4.800 | **..***.*X | 18.098 | | 13.396 | .X | 13.396 | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:01:42,578 | test | WARNING | Thread-8139 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HElIugAGDszfH41yfoB--28-459000:UqqoKzC4wdVyYlJRUOv-28-460000 update failed keys from task: [u'test_collections-3', u'test_collections-4'] 2021-05-03 05:01:42,578 | test | WARNING | Thread-8142 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HElIugAGDszfH41yfoB--28-459000:uvxZa7nZCobqhxiDoE-20-694000 create failed keys from task: [u'test_collections-827', u'test_collections-729', u'test_collections-1050', u'test_collections-1608', u'test_collections-1806', u'test_collections-206', u'test_collections-540', u'test_collections-1523', u'test_collections-1821', u'test_collections-1704', u'test_collections-3628', u'test_collections-2653', u'test_collections-2', u'test_collections-2491', u'test_collections-2501', u'test_collections-1798', u'test_collections-1996', u'test_collections-396', u'test_collections-1371', u'test_collections-3053', u'test_collections-1694'] 2021-05-03 05:01:42,605 | test | WARNING | Thread-8139 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HElIugAGDszfH41yfoB--28-459000:UqqoKzC4wdVyYlJRUOv-28-460000 delete failed keys from task: [u'test_collections-2', u'test_collections-0'] 2021-05-03 05:01:42,828 | test | WARNING | Thread-8303 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:pz0mDCN1wF0J0XZFtn-28-467000 delete failed keys from task: [u'test_collections-48', u'test_collections-62', u'test_collections-357', u'test_collections-31', u'test_collections-312', u'test_collections-165', u'test_collections-130', u'test_collections-182', u'test_collections-150', u'test_collections-282'] 2021-05-03 05:01:42,831 | test | WARNING | Thread-8308 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:yPhauax-28-466000 create failed keys from task: [u'test_collections-2551'] 2021-05-03 05:01:42,832 | test | WARNING | Thread-8310 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:v-28-465000 update failed keys from task: [u'test_collections-836', u'test_collections-666'] 2021-05-03 05:01:42,838 | test | WARNING | Thread-8312 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default update failed keys from task: [u'test_collections-849', u'test_collections-934', u'test_collections-931', u'test_collections-920', u'test_collections-974'] 2021-05-03 05:01:42,839 | test | WARNING | Thread-8318 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F8P_F5H8QOZ-28-467000 update failed keys from task: [u'test_collections-708', u'test_collections-907', u'test_collections-926', u'test_collections-828', u'test_collections-846', u'test_collections-923', u'test_collections-689', u'test_collections-920', u'test_collections-942', u'test_collections-525', u'test_collections-965', u'test_collections-520', u'test_collections-561', u'test_collections-879', u'test_collections-535', u'test_collections-931', u'test_collections-655', u'test_collections-630', u'test_collections-672', u'test_collections-790'] 2021-05-03 05:01:42,841 | test | WARNING | Thread-8322 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:6ww7zVgOfHMKZhF-28-468000 update failed keys from task: [u'test_collections-627', u'test_collections-669', u'test_collections-742', u'test_collections-641', u'test_collections-543'] 2021-05-03 05:01:42,842 | test | WARNING | Thread-8329 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:1Guk0rC8jODFV-28-467000 delete failed keys from task: [u'test_collections-8'] 2021-05-03 05:01:42,844 | test | WARNING | Thread-8330 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:K-KQOucmbVuZz-28-465000 update failed keys from task: [u'test_collections-711', u'test_collections-850'] 2021-05-03 05:01:42,844 | test | WARNING | Thread-8332 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:k3%-19-245000:DePVvM-19-548000 create failed keys from task: [u'test_collections-4261', u'test_collections-3636', u'test_collections-3913', u'test_collections-3936', u'test_collections-3883'] 2021-05-03 05:01:42,891 | test | WARNING | Thread-8443 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:CqbWtD0tt5BgtQN5-28-426000 update failed keys from task: [u'test_collections-4'] 2021-05-03 05:01:42,891 | test | WARNING | Thread-8445 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:E1EtV9VVfZlKs0KEAF3w-28-427000 update failed keys from task: [u'test_collections-4'] 2021-05-03 05:01:42,904 | test | WARNING | Thread-8482 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:RD1q-17-219000:G-KXru6_tVHChXo-17-316000 create failed keys from task: [u'test_collections-4580', u'test_collections-4182', u'test_collections-3988', u'test_collections-3945', u'test_collections-4012', u'test_collections-4473', u'test_collections-4412', u'test_collections-4234', u'test_collections-4410', u'test_collections-3818', u'test_collections-4582', u'test_collections-4643'] 2021-05-03 05:01:42,913 | test | WARNING | Thread-8503 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:ss1t5-zIAB05rSYnjPP-28-441000:m7eHMb82CPP-28-442000 update failed keys from task: [u'test_collections-4'] 2021-05-03 05:01:43,072 | test | WARNING | Thread-8522 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:-%-17-541000:11pz-17-672000 create failed keys from task: [u'test_collections-4291', u'test_collections-4724', u'test_collections-3611', u'test_collections-3781', u'test_collections-3631', u'test_collections-4301'] 2021-05-03 05:01:43,076 | test | WARNING | Thread-8521 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:-%-17-541000:K1h6a6Lnq97DpnC-17-671000 create failed keys from task: [u'test_collections-4580', u'test_collections-4182', u'test_collections-4757', u'test_collections-3988', u'test_collections-3945', u'test_collections-4012', u'test_collections-4473', u'test_collections-4412', u'test_collections-4234', u'test_collections-4410', u'test_collections-3818', u'test_collections-4582'] 2021-05-03 05:01:43,112 | test | WARNING | Thread-8318 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F8P_F5H8QOZ-28-467000 delete failed keys from task: [u'test_collections-38', u'test_collections-127', u'test_collections-487', u'test_collections-100', u'test_collections-82', u'test_collections-142', u'test_collections-384', u'test_collections-3', u'test_collections-214', u'test_collections-256', u'test_collections-93'] 2021-05-03 05:01:43,183 | test | WARNING | Thread-8318 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F8P_F5H8QOZ-28-467000 create failed keys from task: [u'test_collections-2629', u'test_collections-2853', u'test_collections-2731', u'test_collections-2760'] 2021-05-03 05:01:43,573 | test | WARNING | Thread-8333 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:k3%-19-245000:abs-19-641000 create failed keys from task: [u'test_collections-4190', u'test_collections-4581', u'test_collections-4790', u'test_collections-4944', u'test_collections-4000', u'test_collections-4600', u'test_collections-4411'] 2021-05-03 05:01:43,579 | test | WARNING | Thread-8310 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:v-28-465000 delete failed keys from task: [u'test_collections-54', u'test_collections-97', u'test_collections-114', u'test_collections-455', u'test_collections-354', u'test_collections-340', u'test_collections-231', u'test_collections-240'] 2021-05-03 05:01:43,585 | test | WARNING | Thread-8178 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:rw7qxmqS88SkFAt1mU6W-18-198000:0TXCRtUbB46F5LLhoRsO-18-399000 create failed keys from task: [u'test_collections-827', u'test_collections-729', u'test_collections-1050', u'test_collections-1608', u'test_collections-1806', u'test_collections-206', u'test_collections-540', u'test_collections-1523', u'test_collections-1821', u'test_collections-1704', u'test_collections-3628', u'test_collections-2653', u'test_collections-2', u'test_collections-1798', u'test_collections-1996', u'test_collections-396', u'test_collections-1371', u'test_collections-3053', u'test_collections-1694'] 2021-05-03 05:01:42,904 | test | WARNING | Thread-8483 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:RD1q-17-219000:3IdzL%V0oeZG6PL-17-329000 create failed keys from task: [u'test_collections-4580', u'test_collections-4182', u'test_collections-3929', u'test_collections-4849', u'test_collections-4012', u'test_collections-4582', u'test_collections-4473', u'test_collections-4412', u'test_collections-4410', u'test_collections-3972'] 2021-05-03 05:01:43,664 | test | WARNING | Thread-8340 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:OJ0q313-28-475000 update failed keys from task: [u'test_collections-848', u'test_collections-865', u'test_collections-855', u'test_collections-666', u'test_collections-820', u'test_collections-542'] 2021-05-03 05:01:43,690 | test | WARNING | Thread-8634 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:D_N85%xuHOJx0LmI-15-990000:edeWnz8Clvfs77Z3YnC-16-91000 create failed keys from task: [u'test_collections-4580', u'test_collections-4182', u'test_collections-3929', u'test_collections-4757', u'test_collections-4012', u'test_collections-4473', u'test_collections-4412', u'test_collections-4234', u'test_collections-4410', u'test_collections-4582', u'test_collections-3972'] 2021-05-03 05:01:43,690 | test | WARNING | Thread-8635 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:D_N85%xuHOJx0LmI-15-990000:cnMQqDiifhPY-B-16-392000 create failed keys from task: [u'test_collections-4580', u'test_collections-4182', u'test_collections-3929', u'test_collections-4012', u'test_collections-4473', u'test_collections-3465', u'test_collections-4412', u'test_collections-4234', u'test_collections-4410', u'test_collections-4582', u'test_collections-4643', u'test_collections-3972'] 2021-05-03 05:01:43,696 | test | WARNING | Thread-8355 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:Z-28-474000 update failed keys from task: [u'test_collections-705', u'test_collections-524', u'test_collections-964', u'test_collections-643', u'test_collections-621', u'test_collections-644', u'test_collections-564', u'test_collections-740', u'test_collections-839', u'test_collections-533', u'test_collections-530'] 2021-05-03 05:01:43,700 | test | WARNING | Thread-8340 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:OJ0q313-28-475000 delete failed keys from task: [u'test_collections-114', u'test_collections-455', u'test_collections-71', u'test_collections-354', u'test_collections-340', u'test_collections-231', u'test_collections-240'] 2021-05-03 05:01:43,721 | test | WARNING | Thread-8335 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:UGBkg-28-477000 update failed keys from task: [u'test_collections-634', u'test_collections-671'] 2021-05-03 05:01:43,721 | test | WARNING | Thread-8336 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:nF3ARDmo-EVo0-28-474000 create failed keys from task: [u'test_collections-2655'] 2021-05-03 05:01:43,723 | test | WARNING | Thread-8337 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:lEbef7-28-476000 delete failed keys from task: [u'test_collections-209', u'test_collections-399', u'test_collections-189', u'test_collections-184'] 2021-05-03 05:01:43,727 | test | WARNING | Thread-8358 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 update failed keys from task: [u'test_collections-728', u'test_collections-705', u'test_collections-967', u'test_collections-601', u'test_collections-602', u'test_collections-866', u'test_collections-940', u'test_collections-644', u'test_collections-740', u'test_collections-517', u'test_collections-859', u'test_collections-632', u'test_collections-552'] 2021-05-03 05:01:43,729 | test | WARNING | Thread-8335 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:UGBkg-28-477000 delete failed keys from task: [u'test_collections-88', u'test_collections-238', u'test_collections-146', u'test_collections-90', u'test_collections-103', u'test_collections-189', u'test_collections-321'] 2021-05-03 05:01:43,805 | test | WARNING | Thread-8360 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 update failed keys from task: [u'test_collections-829', u'test_collections-907', u'test_collections-846', u'test_collections-923', u'test_collections-920', u'test_collections-942', u'test_collections-965', u'test_collections-561', u'test_collections-637', u'test_collections-655', u'test_collections-630'] 2021-05-03 05:01:43,805 | test | WARNING | Thread-8359 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 update failed keys from task: [u'test_collections-719', u'test_collections-849', u'test_collections-506', u'test_collections-924', u'test_collections-526', u'test_collections-934', u'test_collections-657', u'test_collections-976', u'test_collections-974', u'test_collections-675'] 2021-05-03 05:01:43,805 | test | WARNING | Thread-8439 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:A_-19-946000:FFb5-20-102000 create failed keys from task: [u'test_collections-1800', u'test_collections-1658', u'test_collections-1132', u'test_collections-1990'] 2021-05-03 05:01:43,825 | test | WARNING | Thread-8363 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:yV5HcmKzB3if3-28-476000 update failed keys from task: [u'test_collections-748', u'test_collections-645', u'test_collections-524', u'test_collections-566', u'test_collections-643', u'test_collections-600', u'test_collections-884', u'test_collections-759', u'test_collections-698', u'test_collections-951', u'test_collections-897', u'test_collections-653', u'test_collections-791', u'test_collections-792'] 2021-05-03 05:01:43,836 | test | WARNING | Thread-8404 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:7ygvlc1Li30FxiV-28-472000 update failed keys from task: [u'test_collections-719', u'test_collections-849', u'test_collections-934', u'test_collections-976', u'test_collections-974'] 2021-05-03 05:01:43,854 | test | WARNING | Thread-8370 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:vq-28-469000 update failed keys from task: [u'test_collections-847', u'test_collections-529', u'test_collections-768', u'test_collections-626', u'test_collections-887', u'test_collections-979', u'test_collections-537', u'test_collections-835', u'test_collections-756', u'test_collections-797', u'test_collections-512', u'test_collections-895'] 2021-05-03 05:01:43,861 | test | WARNING | Thread-8372 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:p91eF4no-28-469000 delete failed keys from task: [u'test_collections-496'] 2021-05-03 05:01:43,861 | test | WARNING | Thread-8369 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:E_InIFQkSOujfh-IAS-28-469000 update failed keys from task: [u'test_collections-739', u'test_collections-726', u'test_collections-713', u'test_collections-626', u'test_collections-756', u'test_collections-952', u'test_collections-760', u'test_collections-640', u'test_collections-695'] 2021-05-03 05:01:43,890 | test | WARNING | Thread-8415 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:collection1 delete failed keys from task: [u'test_collections-435', u'test_collections-372'] 2021-05-03 05:01:43,901 | test | WARNING | Thread-8415 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:collection1 create failed keys from task: [u'test_collections-2546'] 2021-05-03 05:01:43,918 | test | WARNING | Thread-8404 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:7ygvlc1Li30FxiV-28-472000 delete failed keys from task: [u'test_collections-439', u'test_collections-469', u'test_collections-448', u'test_collections-247', u'test_collections-369', u'test_collections-213', u'test_collections-252', u'test_collections-383'] 2021-05-03 05:01:43,937 | test | WARNING | Thread-8412 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:pqQlBLVhIRfhPnlG-28-473000 update failed keys from task: [u'test_collections-867', u'test_collections-835', u'test_collections-799', u'test_collections-700', u'test_collections-677', u'test_collections-575', u'test_collections-652', u'test_collections-881', u'test_collections-870'] 2021-05-03 05:01:44,000 | test | WARNING | Thread-8384 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:C-28-470000 update failed keys from task: [u'test_collections-714', u'test_collections-747'] 2021-05-03 05:01:44,006 | test | WARNING | Thread-8358 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:w_DK1jUIhDOr4-28-474000 delete failed keys from task: [u'test_collections-408', u'test_collections-96', u'test_collections-248', u'test_collections-136', u'test_collections-276', u'test_collections-341', u'test_collections-140', u'test_collections-173', u'test_collections-170'] 2021-05-03 05:01:44,032 | test | WARNING | Thread-8426 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:9VB-28-472000 update failed keys from task: [u'test_collections-609', u'test_collections-848', u'test_collections-944', u'test_collections-865', u'test_collections-855', u'test_collections-676', u'test_collections-820', u'test_collections-690'] 2021-05-03 05:01:44,035 | test | WARNING | Thread-8360 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:6OMY%fCXa4ACXQF-28-475000 delete failed keys from task: [u'test_collections-408', u'test_collections-108', u'test_collections-349', u'test_collections-96', u'test_collections-248', u'test_collections-266', u'test_collections-341', u'test_collections-140', u'test_collections-28', u'test_collections-85', u'test_collections-412', u'test_collections-116', u'test_collections-136', u'test_collections-276', u'test_collections-497', u'test_collections-173'] 2021-05-03 05:01:44,042 | test | WARNING | Thread-8359 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 delete failed keys from task: [u'test_collections-94', u'test_collections-203', u'test_collections-324', u'test_collections-214', u'test_collections-125', u'test_collections-378', u'test_collections-384', u'test_collections-393'] 2021-05-03 05:01:44,049 | test | WARNING | Thread-8384 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:C-28-470000 create failed keys from task: [u'test_collections-2502'] 2021-05-03 05:01:44,055 | test | WARNING | Thread-8421 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:GiB-28-472000 update failed keys from task: [u'test_collections-915', u'test_collections-789'] 2021-05-03 05:01:44,085 | test | WARNING | Thread-8421 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:GiB-28-472000 delete failed keys from task: [u'test_collections-495'] 2021-05-03 05:01:44,098 | test | WARNING | Thread-8421 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:GiB-28-472000 create failed keys from task: [u'test_collections-2872'] 2021-05-03 05:01:44,105 | test | WARNING | Thread-8441 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8J1Z2rT-20-281000:wK%cqk3bEJwp1V3-20-491000 create failed keys from task: [u'test_collections-708', u'test_collections-1804', u'test_collections-1488', u'test_collections-560', u'test_collections-1965', u'test_collections-1647', u'test_collections-1518', u'test_collections-878', u'test_collections-459', u'test_collections-478', u'test_collections-698', u'test_collections-1994', u'test_collections-653'] 2021-05-03 05:01:44,105 | test | WARNING | Thread-8442 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8J1Z2rT-20-281000:W-J0er-20-490000 create failed keys from task: [u'test_collections-2095', u'test_collections-2105', u'test_collections-1204', u'test_collections-1656', u'test_collections-1394', u'test_collections-2222', u'test_collections-1672'] 2021-05-03 05:01:44,111 | test | WARNING | Thread-8366 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:12-OWL-28-477000 update failed keys from task: [u'test_collections-905', u'test_collections-603', u'test_collections-566', u'test_collections-688', u'test_collections-740', u'test_collections-760', u'test_collections-561', u'test_collections-917', u'test_collections-615', u'test_collections-650', u'test_collections-892'] 2021-05-03 05:01:44,111 | test | WARNING | Thread-8370 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:vq-28-469000 delete failed keys from task: [u'test_collections-59', u'test_collections-62', u'test_collections-357', u'test_collections-31', u'test_collections-312', u'test_collections-199', u'test_collections-154', u'test_collections-182', u'test_collections-282'] 2021-05-03 05:01:44,112 | test | WARNING | Thread-8381 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collection_1 update failed keys from task: [u'test_collections-682'] 2021-05-03 05:01:44,118 | test | WARNING | Thread-8389 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:eyc8g0SRUfoC3%B-28-469000 update failed keys from task: [u'test_collections-578'] 2021-05-03 05:01:44,118 | test | WARNING | Thread-8381 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collection_1 create failed keys from task: [u'test_collections-2502'] 2021-05-03 05:01:44,119 | test | WARNING | Thread-8401 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:x%ashpqwLrSRhJIqm-28-473000 update failed keys from task: [u'test_collections-803', u'test_collections-914', u'test_collections-603', u'test_collections-615', u'test_collections-965', u'test_collections-688', u'test_collections-784', u'test_collections-796', u'test_collections-650', u'test_collections-561'] 2021-05-03 05:01:44,148 | test | WARNING | Thread-8359 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:V9IuEbS-28-474000:e_f1oW1cR-28-475000 create failed keys from task: [u'test_collections-2609', u'test_collections-2536', u'test_collections-2730'] 2021-05-03 05:01:53,556 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 05:03:01,065 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 05:03:01,065 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 05:03:37,049 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 05:03:37,051 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-03 05:04:05,165 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 05:04:05,167 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 05:04:48,710 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 05:05:06,809 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.1559998989 seconds to complete 2021-05-03 05:05:06,829 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1931.613s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_3 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_4', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 4, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 05:05:16,930 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 05:05:18,010 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 05:05:19,009 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b8e type=client open_count=1 channel=[id: 0x428c5881, 0.0.0.0/0.0.0.0:50605] timeout=300.0>) 2021-05-03 05:05:24,970 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b8f type=client open_count=1 channel=[id: 0x57addbae, 0.0.0.0/0.0.0.0:49713] timeout=300.0>) 2021-05-03 05:05:27,002 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b90 type=client open_count=1 channel=[id: 0x677ebfcc, 0.0.0.0/0.0.0.0:33433] timeout=300.0>) 2021-05-03 05:05:29,036 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b91 type=client open_count=1 channel=[id: 0x2cfdcdc8, 0.0.0.0/0.0.0.0:40003] timeout=300.0>) 2021-05-03 05:05:31,088 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b92 type=client open_count=1 channel=[id: 0x360c9399, 0.0.0.0/0.0.0.0:59649] timeout=300.0>) 2021-05-03 05:05:33,130 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b93 type=client open_count=1 channel=[id: 0xc9db711d, 0.0.0.0/0.0.0.0:37078] timeout=300.0>) 2021-05-03 05:05:35,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b94 type=client open_count=1 channel=[id: 0x111748c6, 0.0.0.0/0.0.0.0:39491] timeout=300.0>) 2021-05-03 05:05:37,210 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b95 type=client open_count=1 channel=[id: 0xcc49de4d, 0.0.0.0/0.0.0.0:52108] timeout=300.0>) 2021-05-03 05:05:39,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x116b96 type=client open_count=1 channel=[id: 0x4c2f9dca, 0.0.0.0/0.0.0.0:49437] timeout=300.0>) 2021-05-03 05:05:41,262 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b97 type=client open_count=1 channel=[id: 0xcd05a4b5, 0.0.0.0/0.0.0.0:44785] timeout=300.0>) 2021-05-03 05:05:49,361 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b98 type=client open_count=1 channel=[id: 0x202e3865, 0.0.0.0/0.0.0.0:49067] timeout=300.0>) 2021-05-03 05:05:51,391 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b99 type=client open_count=1 channel=[id: 0x7d0dcf1e, 0.0.0.0/0.0.0.0:60317] timeout=300.0>) 2021-05-03 05:05:53,418 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b9a type=client open_count=1 channel=[id: 0x69418a97, 0.0.0.0/0.0.0.0:60863] timeout=300.0>) 2021-05-03 05:05:55,444 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b9b type=client open_count=1 channel=[id: 0xe4ae62a3, 0.0.0.0/0.0.0.0:33759] timeout=300.0>) 2021-05-03 05:05:57,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b9c type=client open_count=1 channel=[id: 0x53cea91d, 0.0.0.0/0.0.0.0:57983] timeout=300.0>) 2021-05-03 05:05:59,559 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b9d type=client open_count=1 channel=[id: 0x2e406ea0, 0.0.0.0/0.0.0.0:49077] timeout=300.0>) 2021-05-03 05:06:01,655 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x116b9e type=client open_count=1 channel=[id: 0x5417f3aa, 0.0.0.0/0.0.0.0:40061] timeout=300.0>) 2021-05-03 05:06:03,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116b9f type=client open_count=1 channel=[id: 0x54e351ee, 0.0.0.0/0.0.0.0:42140] timeout=300.0>) 2021-05-03 05:06:12,229 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba0 type=client open_count=1 channel=[id: 0xa552551c, 0.0.0.0/0.0.0.0:53707] timeout=300.0>) 2021-05-03 05:06:14,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba1 type=client open_count=1 channel=[id: 0xb1e35137, 0.0.0.0/0.0.0.0:34465] timeout=300.0>) 2021-05-03 05:06:16,286 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba2 type=client open_count=1 channel=[id: 0xdb4b663c, 0.0.0.0/0.0.0.0:55091] timeout=300.0>) 2021-05-03 05:06:18,338 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba3 type=client open_count=1 channel=[id: 0x9dd10d02, 0.0.0.0/0.0.0.0:43207] timeout=300.0>) 2021-05-03 05:06:20,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba4 type=client open_count=1 channel=[id: 0x4abe300d, 0.0.0.0/0.0.0.0:52731] timeout=300.0>) 2021-05-03 05:06:22,441 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba5 type=client open_count=1 channel=[id: 0xf4a09431, 0.0.0.0/0.0.0.0:57209] timeout=300.0>) 2021-05-03 05:06:24,500 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba6 type=client open_count=1 channel=[id: 0x23a727ee, 0.0.0.0/0.0.0.0:33745] timeout=300.0>) 2021-05-03 05:06:26,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba7 type=client open_count=1 channel=[id: 0x583d2eae, 0.0.0.0/0.0.0.0:51211] timeout=300.0>) 2021-05-03 05:06:28,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x116ba8 type=client open_count=1 channel=[id: 0x37fb654b, 0.0.0.0/0.0.0.0:42333] timeout=300.0>) 2021-05-03 05:06:30,673 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116ba9 type=client open_count=1 channel=[id: 0x280bbd5d, 0.0.0.0/0.0.0.0:36952] timeout=300.0>) 2021-05-03 05:06:39,825 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116baa type=client open_count=1 channel=[id: 0x27a76a8e, 0.0.0.0/0.0.0.0:39377] timeout=300.0>) 2021-05-03 05:06:41,849 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116bab type=client open_count=1 channel=[id: 0x40a04045, 0.0.0.0/0.0.0.0:33663] timeout=300.0>) 2021-05-03 05:06:43,871 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116bac type=client open_count=1 channel=[id: 0x864acbe4, 0.0.0.0/0.0.0.0:38340] timeout=300.0>) 2021-05-03 05:06:45,898 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116bad type=client open_count=1 channel=[id: 0x3dfdc44d, 0.0.0.0/0.0.0.0:47738] timeout=300.0>) 2021-05-03 05:06:47,937 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116bae type=client open_count=1 channel=[id: 0xc1237ae0, 0.0.0.0/0.0.0.0:49875] timeout=300.0>) 2021-05-03 05:06:50,003 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116baf type=client open_count=1 channel=[id: 0xc6e6f218, 0.0.0.0/0.0.0.0:53472] timeout=300.0>) 2021-05-03 05:06:52,051 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116bb0 type=client open_count=1 channel=[id: 0x34908085, 0.0.0.0/0.0.0.0:51351] timeout=300.0>) 2021-05-03 05:06:54,088 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x116bb1 type=client open_count=1 channel=[id: 0x99be2af5, 0.0.0.0/0.0.0.0:51828] timeout=300.0>) 2021-05-03 05:06:56,101 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116bb2 type=client open_count=1 channel=[id: 0x4da631be, 0.0.0.0/0.0.0.0:59305] timeout=300.0>) 2021-05-03 05:07:01,795 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116bb3 type=client open_count=1 channel=[id: 0x00464d0f, 0.0.0.0/0.0.0.0:35872] timeout=300.0>) 2021-05-03 05:07:03,825 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116bb4 type=client open_count=1 channel=[id: 0x63869abe, 0.0.0.0/0.0.0.0:39343] timeout=300.0>) 2021-05-03 05:07:05,845 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116bb5 type=client open_count=1 channel=[id: 0xaf2ba3b5, 0.0.0.0/0.0.0.0:49889] timeout=300.0>) 2021-05-03 05:07:07,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116bb6 type=client open_count=1 channel=[id: 0x4fb6c440, 0.0.0.0/0.0.0.0:57145] timeout=300.0>) 2021-05-03 05:07:09,895 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x116bb7 type=client open_count=1 channel=[id: 0xfb971ef0, 0.0.0.0/0.0.0.0:54887] timeout=300.0>) 2021-05-03 05:07:11,933 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 05:07:28,900 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 05:07:43,648 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 05:07:43,677 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 05:07:43,733 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 05:08:15,506 | test | INFO | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.42479213908 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:08:15,556 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:08:20,608 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:08:30,730 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:08:30,757 | test | INFO | pool-6-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.2509999275 sec 2021-05-03 05:08:31,308 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 05:09:15,368 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 05:09:15,615 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 613055968 | 86886512 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 549442704 | 111371456 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 587352192 | 62087492 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-03 05:09:15,615 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 05:10:36,081 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:10:36,099 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:25:56,417 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **........*..........................................................*............*X | 777.894 | | 676.094 | ...................................................................**.........*..................*.....*X | 992.814 | | 901.114 | .........................................................................................X | 901.114 | | 754.707 | ...........................................................................*X | 754.707 | | 734.107 | ........................................................................*X | 734.107 | | 733.174 | ........................................................................**X | 733.174 | | 725.674 | ........................................................................*....*.*X | 780.510 | | 769.110 | ............................................................................*X | 769.110 | | 716.414 | .......................................................................X | 716.414 | | 638.747 | ...............................................................X | 638.747 | | 609.247 | ............................................................*X | 609.247 | | 580.704 | .........................................................X | 580.704 | | 521.246 | ...................................................*X | 521.246 | | 487.946 | ................................................*X | 487.946 | | 466.895 | ..............................................X | 466.895 | | 430.160 | ..........................................*.**..*.....*.**....*........**....*.*.*X | 702.803 | | 691.332 | ....................................................................**..X | 706.298 | | 672.797 | ..................................................................X | 672.797 | | 652.397 | ................................................................*X | 652.397 | | 569.697 | ........................................................X | 569.697 | | 454.600 | ............................................*X | 454.600 | | 421.700 | .........................................*.*...****..**...**..***.*..*.......**...**.X | 673.177 | | 671.846 | ..................................................................**X | 674.946 | | 624.762 | .............................................................**X | 624.762 | | 612.462 | ............................................................*.*.........**.**X | 718.995 | | 666.295 | ..................................................................*X | 666.295 | | 656.595 | .................................................................X | 656.595 | | 602.388 | ...........................................................X | 602.388 | | 518.988 | ...................................................X | 518.988 | | 473.088 | ..............................................**X | 473.088 | | 467.480 | ..............................................X | 467.480 | | 465.980 | .............................................*.**X | 467.943 | | 425.643 | .........................................X | 425.643 | | 412.350 | ........................................*X | 412.350 | | 406.341 | .......................................**....*...**...*....**..*..*....*X | 616.887 | | 609.387 | ............................................................X | 609.387 | | 582.665 | .........................................................X | 582.665 | | 531.126 | ....................................................*X | 531.126 | | 523.726 | ...................................................X | 523.726 | | 521.300 | ...................................................*X | 521.300 | | 483.400 | ...............................................X | 483.400 | | 446.100 | ...........................................*X | 446.100 | | 442.600 | ...........................................*..*..*X | 485.139 | | 476.139 | ..............................................X | 476.139 | | 445.139 | ...........................................*X | 445.139 | | 437.157 | ...........................................**.*......*..*X | 529.933 | | 529.133 | ....................................................**.*.**..X | 567.471 | | 546.400 | ......................................................**X | 549.500 | | 549.300 | ......................................................**X | 553.900 | | 531.200 | ....................................................*X | 531.200 | | 493.800 | ................................................X | 493.800 | | 449.625 | ............................................*X | 449.625 | | 434.925 | ..........................................X | 434.925 | | 432.125 | ..........................................**.*..**.....*..X | 534.900 | | 529.690 | ....................................................*X | 529.690 | | 492.490 | ................................................*X | 492.490 | | 455.790 | ............................................X | 455.790 | | 448.403 | ............................................*X | 448.403 | | 442.903 | ...........................................*..*...**........*...........**..*X | 699.069 | | 692.269 | ....................................................................X | 692.269 | | 628.100 | ..............................................................*X | 628.100 | | 599.600 | ...........................................................X | 599.600 | | 528.100 | ....................................................*X | 528.100 | | 435.313 | ..........................................X | 435.313 | | 408.213 | ........................................**..*.....**..**.*X | 509.301 | | 504.001 | .................................................*X | 504.001 | | 503.001 | .................................................*.**...**...........**X | 649.112 | | 634.412 | ..............................................................X | 634.412 | | 631.312 | ..............................................................X | 631.312 | | 618.801 | .............................................................**..*X | 640.701 | | 588.901 | ..........................................................X | 588.901 | | 506.600 | ..................................................X | 506.600 | | 454.600 | ............................................**...X | 476.700 | | 472.300 | ..............................................***X | 474.776 | | 454.676 | ............................................*.***...*..***..X | 529.441 | | 524.641 | ...................................................*....**.**X | 569.700 | | 538.200 | .....................................................X | 538.200 | | 534.200 | ....................................................*X | 534.200 | | 516.744 | ...................................................X | 516.744 | | 500.744 | .................................................*X | 500.744 | | 483.344 | ...............................................*X | 483.344 | | 472.943 | ..............................................X | 472.943 | | 440.243 | ...........................................*X | 440.243 | | 437.743 | ...........................................**.*X | 453.800 | | 436.200 | ..........................................*X | 436.200 | | 426.000 | .........................................X | 426.000 | | 396.814 | ......................................**.**.......**...**...*..*.*X | 565.357 | | 552.757 | ......................................................X | 552.757 | | 529.457 | ....................................................*X | 529.457 | | 510.638 | ..................................................**..*.....*...*X | 614.508 | | 591.908 | ..........................................................X | 591.908 | | 514.069 | ..................................................*X | 514.069 | | 444.269 | ...........................................X | 444.269 | | 419.549 | .........................................X | 419.549 | | 373.480 | ....................................**.**.*...**X | 421.036 | | 408.636 | ........................................X | 408.636 | | 391.336 | ......................................*X | 391.336 | | 381.448 | .....................................**.*..*...**....*......**..**...*X | 588.249 | | 576.149 | .........................................................*X | 576.149 | | 518.500 | ...................................................X | 518.500 | | 468.200 | ..............................................*X | 468.200 | | 445.600 | ...........................................*.*.**.**..*..**.X | 524.361 | | 522.061 | ...................................................*..*X | 545.842 | | 544.342 | .....................................................*X | 544.342 | | 541.842 | .....................................................X | 541.842 | | 516.300 | ...................................................X | 516.300 | | 513.900 | ..................................................***.*....*.**X | 570.558 | | 557.096 | .......................................................**.*..*...*X | 615.676 | | 587.776 | ..........................................................*X | 587.776 | | 529.776 | ....................................................X | 529.776 | | 436.578 | ..........................................*X | 436.578 | | 419.478 | .........................................**.****.*...X | 470.994 | | 466.194 | .............................................**...*...**....*X | 562.749 | | 560.488 | .......................................................X | 560.488 | | 557.069 | .......................................................X | 557.069 | | 539.469 | .....................................................*X | 539.469 | | 537.000 | .....................................................*..**.**X | 572.930 | | 539.030 | .....................................................X | 539.030 | | 529.265 | ....................................................*X | 529.265 | | 494.165 | ................................................*X | 494.165 | | 440.965 | ...........................................X | 440.965 | | 412.500 | ........................................**..*.....*..**X | 497.739 | | 494.539 | ................................................X | 494.539 | | 463.955 | .............................................*.*X | 469.255 | | 454.312 | ............................................X | 454.312 | | 453.457 | ............................................X | 453.457 | | 448.257 | ............................................**....**..........*.....**...*...*.....***...*.....X | 828.475 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:25:57,164 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.........*............................X | 43.478 | | 8.700 | ......**X | 8.700 | | 0.000 | ***************.****.X | 2.701 | | 1.900 | **.****X | 3.302 | | 2.402 | .X | 2.402 | | 1.601 | X | 1.601 | | 0.000 | *********.X | 1.500 | | 1.200 | *X | 1.200 | | 0.800 | X | 0.800 | | 0.000 | *********.*.**X | 2.600 | | 1.000 | X | 1.000 | | 0.400 | *X | 0.400 | | 0.000 | ********************************.****.X | 3.001 | | 1.401 | *X | 1.401 | | 0.801 | **.***X | 1.500 | | 1.100 | X | 1.100 | | 0.300 | *X | 0.300 | | 0.000 | *******..***X | 3.000 | | 0.600 | **X | 0.600 | | 0.000 | **********.X | 1.500 | | 0.700 | *X | 0.700 | | 0.400 | *X | 0.400 | | 0.000 | *****.***X | 2.300 | | 0.700 | *X | 0.700 | | 0.400 | *X | 0.400 | | 0.000 | ***************..*.**.......*.............................*....................**.......................**..................X | 116.312 | | 99.489 | ....................................................................................*X | 99.489 | | 63.489 | .....................................................X | 63.489 | | 8.100 | .....*X | 8.100 | | 0.000 | ********************************************************************************************.*..X | 4.500 | | 3.599 | ..*X | 3.599 | | 2.399 | .X | 2.399 | | 0.000 | *********.*****..**X | 4.501 | | 3.301 | .X | 3.301 | | 0.901 | *X | 0.901 | | 0.000 | *****************************...*.....**..*X | 11.700 | | 8.900 | ......X | 8.900 | | 4.100 | ..X | 4.100 | | 2.600 | .**X | 2.600 | | 1.000 | *X | 1.000 | | 0.400 | *X | 0.400 | | 0.000 | *****....*..........*....*X | 21.004 | | 15.404 | ............X | 15.404 | | 4.200 | ..*X | 4.200 | | 0.000 | ***********.*.X | 3.000 | | 2.400 | .*X | 2.400 | | 1.600 | X | 1.600 | | 0.000 | ***.**..X | 4.500 | | 4.200 | ..*X | 4.200 | | 3.800 | ..X | 3.800 | | 3.000 | .*X | 3.000 | | 2.400 | .**X | 2.400 | | 2.400 | .**X | 3.000 | | 2.200 | X | 2.200 | | 0.600 | *X | 0.600 | | 0.000 | *****************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:25:57,313 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **....*...........X | 5.997 | | 4.397 | ..........X | 4.397 | | 1.200 | ..X | 1.200 | | 0.000 | *****************************..*....*..*X | 3.300 | | 2.900 | ......X | 2.900 | | 2.100 | ....*X | 2.100 | | 2.100 | ....*..*....*X | 4.500 | | 3.600 | ........X | 3.600 | | 2.400 | .....X | 2.400 | | 1.600 | ...**.**....***X | 3.800 | | 2.600 | .....X | 2.600 | | 2.600 | .....****.*...*..**..X | 5.799 | | 4.399 | ..........*X | 4.399 | | 2.900 | ......*X | 2.900 | | 2.400 | .....**..*X | 3.300 | | 2.900 | ......X | 2.900 | | 2.500 | .....X | 2.500 | | 1.900 | ...**X | 1.900 | | 1.900 | ...**X | 1.900 | | 0.700 | **X | 0.700 | | 0.400 | *....*..........**....*X | 7.500 | | 1.500 | ..**X | 1.500 | | 0.000 | *.*..***X | 1.500 | | 1.100 | .X | 1.100 | | 0.300 | X | 0.300 | | 0.000 | ****************...***.*X | 1.801 | | 1.800 | ...*X | 1.800 | | 1.200 | ..X | 1.200 | | 0.800 | .*X | 0.800 | | 0.000 | ***..****.X | 1.500 | | 0.700 | *X | 0.700 | | 0.400 | *X | 0.400 | | 0.000 | *****..***.X | 1.500 | | 0.700 | *X | 0.700 | | 0.400 | ***..**X | 1.500 | | 1.100 | .*X | 1.100 | | 0.300 | X | 0.300 | | 0.000 | ******.*..**.**...*......*.................**..................*..................................*X | 32.403 | | 25.197 | ...............................................................*X | 25.197 | | 16.797 | .........................................X | 16.797 | | 0.000 | **********************..................*.....................................*..............*X | 26.998 | | 19.798 | .................................................X | 19.798 | | 5.400 | ............*X | 5.400 | | 0.000 | ********************************************..***.X | 1.500 | | 0.700 | *X | 0.700 | | 0.400 | *X | 0.400 | | 0.000 | *....**.*X | 2.201 | | 1.400 | ..*X | 1.400 | | 0.800 | .*X | 0.800 | | 0.000 | *....**.........**............................*X | 16.399 | | 12.798 | ...............................**......*.................**..............X | 26.999 | | 19.804 | .................................................X | 19.804 | | 5.400 | ............*X | 5.400 | | 0.000 | **................*.................................*............***.......**.................**...............*X | 39.119 | | 30.400 | ............................................................................*X | 30.400 | | 25.199 | ...............................................................X | 25.199 | | 12.499 | ..............................*X | 12.499 | | 9.499 | .......................*X | 9.499 | | 6.701 | ................***.....X | 8.801 | | 7.800 | ..................X | 7.800 | | 5.400 | ............*X | 5.400 | | 5.100 | ............**...*....*X | 8.200 | | 6.700 | ................*X | 6.700 | | 4.700 | ...........X | 4.700 | | 0.700 | X | 0.700 | | 0.400 | *X | 0.400 | | 0.000 | *****.........***..X | 4.500 | | 3.700 | ........X | 3.700 | | 2.100 | ....*X | 2.100 | | 1.200 | ..*X | 1.200 | | 0.800 | .X | 0.800 | | 0.000 | **************************...*X | 1.500 | | 1.200 | ..*X | 1.200 | | 0.800 | .X | 0.800 | | 0.000 | *************....***.*..X | 3.000 | | 1.400 | ..*X | 1.400 | | 0.800 | .*X | 0.800 | | 0.000 | ***************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 05:25:57,801 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 60.8040201005 | 3693387776 | 2469785600 | 0 / 3909087232 | 126000 / 253784 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 48.5177342509 | 3693387776 | 2535989248 | 0 / 3909087232 | 125925 / 254759 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 59.4693028096 | 6078930944 | 4777127936 | 0 / 3758092288 | 126100 / 245453 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 47.9782270607 | 4201836544 | 3079856128 | 0 / 3758092288 | 125689 / 253551 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 05:26:12,263 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 05:28:13,617 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155326032 | 245227540 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 155355984 | 280665857 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 435467040 | 450554218 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-03 05:28:13,618 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-05-03 05:28:23,473 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_in_out 2021-05-03 05:28:23,473 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_in_out 2021-05-03 05:28:31,348 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:28:31,681 | test | INFO | pool-6-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 21.0887814805 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 21.8324078748 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 11.6431451613 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 0.727363932782 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 13.3912599029 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-03 05:28:31,740 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:28:37,964 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:28:42,559 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:28:44,275 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 1.2 2021-05-03 05:28:49,473 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 3.97 2021-05-03 05:28:54,578 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 5.78 2021-05-03 05:28:59,734 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 12.31 2021-05-03 05:29:04,855 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 17.94 2021-05-03 05:29:10,028 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 23.11 2021-05-03 05:29:15,207 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 29.01 2021-05-03 05:29:16,460 | test | INFO | pool-6-thread-9 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:cLr2-40-877000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-368 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:16,461 | test | INFO | pool-6-thread-9 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:cLr2-40-877000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-368 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:16,473 | test | INFO | pool-6-thread-18 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:TVq0J6EUAOQm0-40-876000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-368 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:16,474 | test | INFO | pool-6-thread-18 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:TVq0J6EUAOQm0-40-876000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-368 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:19,584 | test | INFO | pool-6-thread-18 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:TVq0J6EUAOQm0-40-876000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:19,585 | test | INFO | pool-6-thread-18 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:TVq0J6EUAOQm0-40-876000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:19,608 | test | INFO | pool-6-thread-9 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:cLr2-40-877000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:19,608 | test | INFO | pool-6-thread-9 | [table_view:display:72] Keys failed in bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:cLr2-40-877000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-461 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:29:20,401 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 05:29:25,525 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 05:29:30,753 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 05:29:37,275 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 39.84 2021-05-03 05:29:42,496 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 46.04 2021-05-03 05:29:48,266 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 52.2 2021-05-03 05:29:53,542 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 58.68 2021-05-03 05:29:58,851 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 64.69 2021-05-03 05:30:03,997 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 05:30:09,171 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 05:30:14,338 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 68.52 2021-05-03 05:30:20,117 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 74.53 2021-05-03 05:30:24,183 | test | INFO | pool-6-thread-29 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:e-31-520000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-924 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:24,276 | test | INFO | pool-6-thread-16 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:5AqcHVK6vkMA-31-519000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:25,349 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.47 2021-05-03 05:30:25,911 | test | INFO | pool-6-thread-12 | [table_view:display:72] Keys failed in default:_default:0f-v5yPd-31-505000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2678 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:25,913 | test | INFO | pool-6-thread-12 | [table_view:display:72] Keys failed in default:_default:0f-v5yPd-31-505000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:30,563 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 84.48 2021-05-03 05:30:34,693 | test | INFO | pool-6-thread-16 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:5AqcHVK6vkMA-31-519000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-495 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:35,825 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 90.84 2021-05-03 05:30:39,782 | test | INFO | pool-6-thread-23 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:LEuHF-fyMLkXSqfOg3-31-520000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2835 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:39,783 | test | INFO | pool-6-thread-23 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:LEuHF-fyMLkXSqfOg3-31-520000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2835 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:40,974 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 96.21 2021-05-03 05:30:41,296 | test | INFO | pool-6-thread-28 | [table_view:display:72] Keys failed in default:YDSb4Xwa-41-53000:v1Vmg2J-41-222000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1532 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:41,296 | test | INFO | pool-6-thread-28 | [table_view:display:72] Keys failed in default:YDSb4Xwa-41-53000:v1Vmg2J-41-222000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1532 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:41,302 | test | INFO | pool-6-thread-10 | [table_view:display:72] Keys failed in default:YDSb4Xwa-41-53000:MY8lrfcVMAl-Lvo3-41-221000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1532 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:41,303 | test | INFO | pool-6-thread-10 | [table_view:display:72] Keys failed in default:YDSb4Xwa-41-53000:MY8lrfcVMAl-Lvo3-41-221000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1532 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:44,092 | test | INFO | pool-6-thread-4 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:3Z%AkaS-31-518000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-964 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:46,144 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:30:51,288 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:30:54,875 | test | INFO | pool-6-thread-13 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:n-31-518000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-574 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:54,884 | test | INFO | pool-6-thread-29 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:NNlIo-31-519000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-580 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:54,891 | test | INFO | pool-6-thread-21 | [table_view:display:72] Keys failed in default:unQ5aNR-CFj%R4r8EKDG-31-516000:h5R0we-31-517000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-574 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:30:58,039 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:31:08,204 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 05:31:08,206 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 05:31:08,220 | test | INFO | pool-6-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:31:08,434 | test | INFO | pool-6-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 156.753999949 sec 2021-05-03 05:37:02,835 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 05:37:02,838 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 0 to 5000 2021-05-03 05:37:06,805 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 05:37:06,806 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 05:37:06,809 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 3 docs from 0 to 500 2021-05-03 05:37:06,809 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:37:06,811 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 05:37:06,812 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 05:37:06,815 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 05:37:06,816 | test | ERROR | pool-6-thread-6 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 05:39:33,032 | test | INFO | pool-6-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..................*.........................................X | 457.954 | | 457.954 | ..........................................................*................................X | 704.131 | | 673.431 | ......................................................................................X | 673.431 | | 375.687 | ...............................................X | 375.687 | | 129.511 | ...............**.........................................**....................*X | 592.000 | | 348.500 | ............................................**.**.....................................................*X | 765.445 | | 742.045 | ...............................................................................................X | 742.045 | | 658.145 | ....................................................................................X | 658.145 | | 526.602 | ...................................................................X | 526.602 | | 489.202 | ..............................................................*X | 489.202 | | 432.602 | .......................................................X | 432.602 | | 317.641 | ........................................X | 317.641 | | 280.641 | ...................................*X | 280.641 | | 277.253 | ...................................**..***.....X | 332.233 | | 328.633 | .........................................*X | 328.633 | | 298.645 | .....................................X | 298.645 | | 298.645 | .....................................*.....*.......*...X | 410.363 | | 383.263 | ................................................*X | 383.263 | | 277.099 | ...................................*X | 277.099 | | 263.547 | .................................*X | 263.547 | | 263.347 | .................................X | 263.347 | | 232.766 | .............................X | 232.766 | | 218.763 | ...........................**.....**...**........*.....**..**.*.....**.X | 448.767 | | 408.396 | ....................................................X | 408.396 | | 346.200 | ............................................*X | 346.200 | | 301.500 | ......................................*X | 301.500 | | 283.100 | ...................................X | 283.100 | | 252.500 | ...............................**..*X | 264.500 | | 223.577 | ............................**..**.*........*X | 312.136 | | 246.136 | ...............................*X | 246.136 | | 241.736 | ..............................*....*.........**X | 345.000 | | 223.900 | ............................*X | 223.900 | | 199.000 | ........................**...*..............X | 324.924 | | 301.224 | ......................................*X | 301.224 | | 278.854 | ...................................X | 278.854 | | 239.563 | ..............................X | 239.563 | | 239.563 | ..............................*.....*X | 277.963 | | 275.450 | ..................................X | 275.450 | | 244.800 | ..............................*X | 244.800 | | 186.100 | .......................X | 186.100 | | 148.900 | ..................*....**......**.......X | 279.300 | | 227.700 | ............................*X | 227.700 | | 189.800 | .......................X | 189.800 | | 137.200 | ................X | 137.200 | | 89.800 | ..........**....**......*..............*X | 271.700 | | 262.200 | .................................*.*....**..*X | 316.947 | | 304.415 | ......................................X | 304.415 | | 283.013 | ...................................*X | 283.013 | | 280.513 | ...................................X | 280.513 | | 268.071 | .................................X | 268.071 | | 207.926 | ..........................*X | 207.926 | | 206.826 | .........................**....*......X | 282.300 | | 268.200 | .................................*X | 268.200 | | 243.900 | ..............................X | 243.900 | | 185.200 | .......................**...**...*..................**...*..X | 413.838 | | 326.427 | .........................................*X | 326.427 | | 292.708 | .....................................**.....................**..........*X | 532.570 | | 476.170 | .............................................................X | 476.170 | | 406.872 | ....................................................*X | 406.872 | | 347.816 | ............................................X | 347.816 | | 341.316 | ...........................................*X | 341.316 | | 293.716 | .....................................X | 293.716 | | 277.400 | ...................................**.*.**....*X | 324.100 | | 320.100 | ........................................X | 320.100 | | 315.900 | ........................................X | 315.900 | | 270.600 | ..................................*X | 270.600 | | 232.300 | .............................*..*....**.........X | 346.284 | | 321.734 | ........................................*X | 321.734 | | 319.366 | ........................................*X | 319.366 | | 317.688 | ........................................***..*....*...*X | 390.100 | | 379.000 | ................................................*X | 379.000 | | 340.400 | ...........................................X | 340.400 | | 257.941 | ................................X | 257.941 | | 228.625 | ............................*X | 228.625 | | 218.625 | ...........................*.....**..**..X | 284.779 | | 263.993 | .................................*..***....*........*...**..X | 408.117 | | 400.899 | ...................................................*X | 400.899 | | 357.077 | .............................................*X | 357.077 | | 287.444 | ....................................***.....*.....*.........*X | 435.665 | | 417.165 | .....................................................*X | 417.165 | | 380.703 | ................................................**..**.X | 403.642 | | 382.321 | ................................................*X | 382.321 | | 382.021 | ................................................*X | 382.021 | | 350.400 | ............................................**X | 350.400 | | 308.800 | .......................................X | 308.800 | | 297.300 | .....................................X | 297.300 | | 255.100 | ................................**.**.*.....*..**.*X | 334.626 | | 327.026 | .........................................*..**........*.....................*.........**...X | 654.800 | | 642.400 | ..................................................................................*X | 642.400 | | 639.300 | ..................................................................................**.................**X | 767.400 | | 762.800 | ..................................................................................................X | 762.800 | | 688.000 | ........................................................................................X | 688.000 | | 567.300 | ........................................................................X | 567.300 | | 549.741 | ......................................................................**.*.......*X | 608.341 | | 589.352 | ...........................................................................*X | 589.352 | | 570.752 | .........................................................................X | 570.752 | | 499.852 | ................................................................**...*....X | 554.300 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:39:33,459 | test | INFO | pool-6-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******...**.....*.*.........***........*.....................**.................*............*.......................X | 178.761 | | 170.061 | ..............................................................................................X | 170.061 | | 150.661 | ...................................................................................X | 150.661 | | 117.020 | ................................................................*....**.............*X | 148.173 | | 139.573 | .............................................................................X | 139.573 | | 134.073 | ..........................................................................X | 134.073 | | 115.420 | ...............................................................**...*.....*.....................**...X | 172.968 | | 138.829 | ............................................................................*X | 138.829 | | 124.229 | ....................................................................*.*...........*........*X | 160.000 | | 149.100 | ..................................................................................X | 149.100 | | 120.492 | ..................................................................X | 120.492 | | 110.268 | ............................................................**....*.............**.......**..*X | 156.460 | | 147.820 | .................................................................................*X | 147.820 | | 142.602 | ..............................................................................X | 142.602 | | 120.678 | ..................................................................*X | 120.678 | | 112.590 | .............................................................*X | 112.590 | | 109.674 | ............................................................**........**..*................*X | 156.900 | | 154.000 | .....................................................................................**...X | 159.300 | | 156.300 | ......................................................................................X | 156.300 | | 155.400 | .....................................................................................*X | 155.400 | | 150.100 | ..................................................................................X | 150.100 | | 118.589 | .................................................................*X | 118.589 | | 108.389 | ...........................................................*X | 108.389 | | 88.689 | ................................................*..X | 92.115 | | 86.915 | ...............................................**.....X | 96.015 | | 85.800 | ..............................................**.*X | 86.100 | | 71.300 | ......................................X | 71.300 | | 64.300 | ..................................*X | 64.300 | | 59.200 | ................................*...........*..........**.........*X | 113.273 | | 113.185 | ..............................................................**X | 113.924 | | 103.624 | ........................................................X | 103.624 | | 88.418 | ................................................X | 88.418 | | 79.700 | ...........................................**...*........*.**...*X | 105.500 | | 95.800 | ....................................................*......*X | 106.100 | | 96.100 | ....................................................X | 96.100 | | 91.000 | .................................................X | 91.000 | | 69.100 | .....................................*X | 69.100 | | 64.300 | ..................................**.......*.........**......**....*..*......*X | 123.486 | | 104.178 | .........................................................X | 104.178 | | 100.107 | .......................................................*X | 100.107 | | 90.107 | .................................................*........*.......*X | 116.315 | | 111.415 | .............................................................*X | 111.415 | | 69.300 | .....................................**X | 69.300 | | 59.900 | ................................X | 59.900 | | 44.700 | ........................X | 44.700 | | 33.100 | .................*X | 33.100 | | 22.900 | ...........*X | 22.900 | | 18.700 | .........X | 18.700 | | 4.600 | .**X | 4.600 | | 3.900 | .*X | 3.900 | | 2.699 | X | 2.699 | | 2.499 | **.*.**X | 6.803 | | 4.803 | .X | 4.803 | | 3.200 | *X | 3.200 | | 2.300 | *****.X | 3.700 | | 2.200 | X | 2.200 | | 1.800 | *X | 1.800 | | 1.500 | ****X | 1.700 | | 1.200 | X | 1.200 | | 0.900 | X | 0.900 | | 0.600 | ***.**..**X | 6.499 | | 5.899 | ..X | 5.899 | | 4.801 | .*X | 4.801 | | 3.601 | .*X | 3.601 | | 3.000 | ***.X | 3.600 | | 2.600 | X | 2.600 | | 1.600 | *X | 1.600 | | 0.900 | *****.*X | 2.601 | | 1.800 | *X | 1.800 | | 1.100 | *X | 1.100 | | 0.700 | X | 0.700 | | 0.300 | ***X | 0.300 | | 0.100 | ****.*..**.**..X | 11.399 | | 10.602 | ....X | 10.602 | | 8.702 | ...*X | 8.702 | | 4.802 | .X | 4.802 | | 2.099 | *X | 2.099 | | 1.299 | **.****.*X | 4.200 | | 4.200 | .X | 4.200 | | 3.700 | .*X | 3.700 | | 2.900 | X | 2.900 | | 1.800 | X | 1.800 | | 1.600 | *X | 1.600 | | 1.200 | X | 1.200 | | 0.900 | ****....***.*X | 10.701 | | 9.301 | ....*X | 9.301 | | 7.800 | ...*X | 7.800 | | 5.900 | ..*.*.**.X | 11.500 | | 6.400 | ..*X | 6.400 | | 3.100 | **.***.X | 5.999 | | 5.499 | ..X | 5.499 | | 4.399 | .****.*.X | 7.501 | | 7.100 | ..*X | 7.100 | | 5.100 | .X | 5.100 | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:39:34,792 | test | INFO | pool-6-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***********************************X | 0.200 | | 0.100 | X | 0.100 | | 0.000 | *********X | 0.100 | | 0.000 | ***************X | 0.901 | | 0.701 | *X | 0.701 | | 0.401 | X | 0.401 | | 0.000 | ***.*..****.....*...........**............**............*....................*.......*X | 113.812 | | 103.712 | ...............................................................X | 103.712 | | 96.800 | ..........................................................*X | 96.800 | | 86.800 | ....................................................*X | 86.800 | | 86.700 | ....................................................*..*......**.......X | 110.194 | | 106.580 | ................................................................*X | 106.580 | | 96.580 | ..........................................................*X | 96.580 | | 86.590 | ....................................................*....**......X | 103.425 | | 103.421 | ..............................................................X | 103.421 | | 91.400 | .......................................................*X | 91.400 | | 81.400 | .................................................*......*............**......**X | 120.600 | | 104.800 | ...............................................................*X | 104.800 | | 94.800 | .........................................................*X | 94.800 | | 84.800 | ...................................................X | 84.800 | | 82.200 | .................................................***X | 82.400 | | 81.500 | .................................................*X | 81.500 | | 71.600 | ...........................................*X | 71.600 | | 51.600 | ..............................X | 51.600 | | 10.700 | .....X | 10.700 | | 0.600 | **...*.........**......*.......*......**...**.*.**...**.*X | 66.300 | | 61.397 | ....................................**...*............**......*...*.*X | 101.568 | | 101.563 | .............................................................**X | 101.758 | | 99.739 | ............................................................*X | 99.739 | | 99.734 | ............................................................X | 99.734 | | 99.534 | ............................................................X | 99.534 | | 97.495 | ...........................................................****.X | 99.205 | | 98.405 | ...........................................................X | 98.405 | | 93.100 | ........................................................*X | 93.100 | | 86.900 | ....................................................X | 86.900 | | 76.400 | ..............................................****....**X | 82.900 | | 82.500 | .................................................X | 82.500 | | 77.100 | ..............................................X | 77.100 | | 73.100 | ............................................*X | 73.100 | | 63.998 | ......................................*...X | 68.798 | | 62.781 | .....................................**.............**..........*.............................*X | 146.036 | | 145.030 | ........................................................................................X | 145.030 | | 135.030 | ..................................................................................X | 135.030 | | 106.200 | ................................................................*X | 106.200 | | 104.700 | ...............................................................**................***.**X | 132.756 | | 126.929 | .............................................................................X | 126.929 | | 125.636 | ............................................................................**.......*......**.......*X | 157.408 | | 151.160 | ............................................................................................*****......**.*X | 161.936 | | 119.682 | ........................................................................**X | 119.682 | | 91.782 | .......................................................*.X | 93.389 | | 81.715 | .................................................**.....*X | 90.015 | | 85.903 | ....................................................X | 85.903 | | 84.658 | ...................................................X | 84.658 | | 83.358 | ..................................................**..*.*...*X | 93.653 | | 91.130 | .......................................................*X | 91.130 | | 87.900 | .....................................................X | 87.900 | | 82.400 | .................................................*X | 82.400 | | 75.400 | .............................................*.X | 76.200 | | 71.192 | ..........................................**...X | 76.092 | | 70.992 | ..........................................**.*X | 71.505 | | 44.805 | ..........................**X | 44.805 | | 34.201 | ....................X | 34.201 | | 28.501 | ................X | 28.501 | | 14.501 | .......*X | 14.501 | | 10.300 | .....X | 10.300 | | 8.100 | ....**.*...*X | 15.100 | | 8.800 | ....**X | 8.800 | | 4.500 | .X | 4.500 | | 3.200 | *..*..**X | 9.198 | | 8.198 | ....*X | 8.198 | | 6.201 | ..**..**.*X | 9.900 | | 7.000 | ...X | 7.000 | | 3.700 | .X | 3.700 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:39:34,851 | test | WARNING | Thread-11023 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:0f-v5yPd-31-505000 create failed keys from task: [u'test_collections-2678'] 2021-05-03 05:39:34,865 | test | WARNING | Thread-11062 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:YDSb4Xwa-41-53000:v1Vmg2J-41-222000 create failed keys from task: [u'test_collections-1532'] 2021-05-03 05:39:34,865 | test | WARNING | Thread-11063 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:YDSb4Xwa-41-53000:MY8lrfcVMAl-Lvo3-41-221000 create failed keys from task: [u'test_collections-1532'] 2021-05-03 05:39:34,865 | test | WARNING | Thread-11065 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:unQ5aNR-CFj%R4r8EKDG-31-516000:h5R0we-31-517000 update failed keys from task: [u'test_collections-574'] 2021-05-03 05:39:34,865 | test | WARNING | Thread-11067 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:unQ5aNR-CFj%R4r8EKDG-31-516000:NNlIo-31-519000 update failed keys from task: [u'test_collections-580'] 2021-05-03 05:39:34,867 | test | WARNING | Thread-11068 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:unQ5aNR-CFj%R4r8EKDG-31-516000:3Z%AkaS-31-518000 update failed keys from task: [u'test_collections-964'] 2021-05-03 05:39:34,868 | test | WARNING | Thread-11075 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:unQ5aNR-CFj%R4r8EKDG-31-516000:5AqcHVK6vkMA-31-519000 delete failed keys from task: [u'test_collections-203', u'test_collections-495', u'test_collections-393'] 2021-05-03 05:39:34,867 | test | WARNING | Thread-11071 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:unQ5aNR-CFj%R4r8EKDG-31-516000:LEuHF-fyMLkXSqfOg3-31-520000 create failed keys from task: [u'test_collections-2835'] 2021-05-03 05:39:35,184 | test | WARNING | Thread-11328 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:TVq0J6EUAOQm0-40-876000 create failed keys from task: [u'test_collections-368', u'test_collections-461'] 2021-05-03 05:39:35,184 | test | WARNING | Thread-11327 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:R2as7xW6Vx%wNilf7%I2-40-759000:cLr2-40-877000 create failed keys from task: [u'test_collections-368', u'test_collections-461'] 2021-05-03 05:39:35,186 | test | WARNING | Thread-11076 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:unQ5aNR-CFj%R4r8EKDG-31-516000:n-31-518000 update failed keys from task: [u'test_collections-574'] 2021-05-03 05:39:35,184 | test | WARNING | Thread-11077 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:unQ5aNR-CFj%R4r8EKDG-31-516000:e-31-520000 update failed keys from task: [u'test_collections-924'] 2021-05-03 05:39:48,772 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 05:41:19,604 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 05:41:19,604 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 05:41:50,463 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 05:41:50,464 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-03 05:42:21,778 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 05:42:21,779 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 05:42:59,635 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 05:43:17,765 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2000000477 seconds to complete 2021-05-03 05:43:17,779 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2290.853s During the test, Remote Connections: 162, Disconnections: 162 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_4 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_5', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 5, 'bulk_api_crud': 'True'} test_rebalance_cycles (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 05:43:28,043 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #5 test_rebalance_cycles ========= 2021-05-03 05:43:29,211 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #5 test_rebalance_cycles ========= 2021-05-03 05:43:30,303 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a35 type=client open_count=1 channel=[id: 0x364f4eab, 0.0.0.0/0.0.0.0:41995] timeout=300.0>) 2021-05-03 05:43:35,920 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a36 type=client open_count=1 channel=[id: 0x70bd0aa5, 0.0.0.0/0.0.0.0:60943] timeout=300.0>) 2021-05-03 05:43:37,947 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a37 type=client open_count=1 channel=[id: 0x8aa52d32, 0.0.0.0/0.0.0.0:56421] timeout=300.0>) 2021-05-03 05:43:39,979 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a38 type=client open_count=1 channel=[id: 0xee5dc903, 0.0.0.0/0.0.0.0:39122] timeout=300.0>) 2021-05-03 05:43:42,015 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a39 type=client open_count=1 channel=[id: 0xdf3edb78, 0.0.0.0/0.0.0.0:35873] timeout=300.0>) 2021-05-03 05:43:44,081 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a3a type=client open_count=1 channel=[id: 0x983a0d69, 0.0.0.0/0.0.0.0:47601] timeout=300.0>) 2021-05-03 05:43:46,132 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a3b type=client open_count=1 channel=[id: 0x4fdc7207, 0.0.0.0/0.0.0.0:36459] timeout=300.0>) 2021-05-03 05:43:48,154 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a3c type=client open_count=1 channel=[id: 0xf916cb4e, 0.0.0.0/0.0.0.0:48977] timeout=300.0>) 2021-05-03 05:43:50,181 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x173a3d type=client open_count=1 channel=[id: 0x9a09eda4, 0.0.0.0/0.0.0.0:39958] timeout=300.0>) 2021-05-03 05:43:52,204 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a3e type=client open_count=1 channel=[id: 0x8260bed0, 0.0.0.0/0.0.0.0:43477] timeout=300.0>) 2021-05-03 05:44:00,236 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a3f type=client open_count=1 channel=[id: 0xbc7326e7, 0.0.0.0/0.0.0.0:44883] timeout=300.0>) 2021-05-03 05:44:02,278 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a40 type=client open_count=1 channel=[id: 0x7afeeec4, 0.0.0.0/0.0.0.0:33549] timeout=300.0>) 2021-05-03 05:44:04,380 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a41 type=client open_count=1 channel=[id: 0x8387f7bb, 0.0.0.0/0.0.0.0:53705] timeout=300.0>) 2021-05-03 05:44:06,436 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a42 type=client open_count=1 channel=[id: 0x09237670, 0.0.0.0/0.0.0.0:54107] timeout=300.0>) 2021-05-03 05:44:08,529 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a43 type=client open_count=1 channel=[id: 0x7cd0485b, 0.0.0.0/0.0.0.0:34181] timeout=300.0>) 2021-05-03 05:44:10,551 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a44 type=client open_count=1 channel=[id: 0x018573e9, 0.0.0.0/0.0.0.0:57151] timeout=300.0>) 2021-05-03 05:44:12,579 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a45 type=client open_count=1 channel=[id: 0x0e01c80e, 0.0.0.0/0.0.0.0:37103] timeout=300.0>) 2021-05-03 05:44:14,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x173a46 type=client open_count=1 channel=[id: 0x3f6c1e7b, 0.0.0.0/0.0.0.0:47439] timeout=300.0>) 2021-05-03 05:44:16,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a47 type=client open_count=1 channel=[id: 0xba3e13a3, 0.0.0.0/0.0.0.0:44003] timeout=300.0>) 2021-05-03 05:44:25,374 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a48 type=client open_count=1 channel=[id: 0x88783563, 0.0.0.0/0.0.0.0:49583] timeout=300.0>) 2021-05-03 05:44:27,403 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a49 type=client open_count=1 channel=[id: 0x0b1b5efa, 0.0.0.0/0.0.0.0:53375] timeout=300.0>) 2021-05-03 05:44:29,500 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a4a type=client open_count=1 channel=[id: 0x5abb0c0a, 0.0.0.0/0.0.0.0:44529] timeout=300.0>) 2021-05-03 05:44:31,637 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a4b type=client open_count=1 channel=[id: 0x882ab0c0, 0.0.0.0/0.0.0.0:38017] timeout=300.0>) 2021-05-03 05:44:33,696 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a4c type=client open_count=1 channel=[id: 0x6125aec8, 0.0.0.0/0.0.0.0:56941] timeout=300.0>) 2021-05-03 05:44:35,729 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a4d type=client open_count=1 channel=[id: 0xb380323a, 0.0.0.0/0.0.0.0:50719] timeout=300.0>) 2021-05-03 05:44:37,802 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a4e type=client open_count=1 channel=[id: 0x83ef40fa, 0.0.0.0/0.0.0.0:55185] timeout=300.0>) 2021-05-03 05:44:39,826 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a4f type=client open_count=1 channel=[id: 0xd12c911f, 0.0.0.0/0.0.0.0:47307] timeout=300.0>) 2021-05-03 05:44:41,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a50 type=client open_count=1 channel=[id: 0x18e5af5b, 0.0.0.0/0.0.0.0:34339] timeout=300.0>) 2021-05-03 05:44:43,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x173a51 type=client open_count=1 channel=[id: 0x5b8eac7e, 0.0.0.0/0.0.0.0:39225] timeout=300.0>) 2021-05-03 05:44:45,927 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a52 type=client open_count=1 channel=[id: 0x691931bf, 0.0.0.0/0.0.0.0:56283] timeout=300.0>) 2021-05-03 05:44:54,990 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a53 type=client open_count=1 channel=[id: 0x0e8474bf, 0.0.0.0/0.0.0.0:39932] timeout=300.0>) 2021-05-03 05:44:57,012 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a54 type=client open_count=1 channel=[id: 0xd57da177, 0.0.0.0/0.0.0.0:43493] timeout=300.0>) 2021-05-03 05:44:59,036 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a55 type=client open_count=1 channel=[id: 0x5f51beaa, 0.0.0.0/0.0.0.0:45147] timeout=300.0>) 2021-05-03 05:45:01,059 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a56 type=client open_count=1 channel=[id: 0x9dd59da8, 0.0.0.0/0.0.0.0:45240] timeout=300.0>) 2021-05-03 05:45:03,079 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a57 type=client open_count=1 channel=[id: 0x6c2f3412, 0.0.0.0/0.0.0.0:57575] timeout=300.0>) 2021-05-03 05:45:05,098 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a58 type=client open_count=1 channel=[id: 0x2fa8f219, 0.0.0.0/0.0.0.0:53753] timeout=300.0>) 2021-05-03 05:45:07,157 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a59 type=client open_count=1 channel=[id: 0x8cdab3a8, 0.0.0.0/0.0.0.0:33789] timeout=300.0>) 2021-05-03 05:45:09,206 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a5a type=client open_count=1 channel=[id: 0xb75c5523, 0.0.0.0/0.0.0.0:34757] timeout=300.0>) 2021-05-03 05:45:11,229 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x173a5b type=client open_count=1 channel=[id: 0xceb00898, 0.0.0.0/0.0.0.0:48194] timeout=300.0>) 2021-05-03 05:45:13,313 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173a5c type=client open_count=1 channel=[id: 0x6e97ee1c, 0.0.0.0/0.0.0.0:47447] timeout=300.0>) 2021-05-03 05:45:18,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173a5d type=client open_count=1 channel=[id: 0xad19852d, 0.0.0.0/0.0.0.0:45387] timeout=300.0>) 2021-05-03 05:45:20,911 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173a5e type=client open_count=1 channel=[id: 0x8cbec22f, 0.0.0.0/0.0.0.0:44891] timeout=300.0>) 2021-05-03 05:45:22,933 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173a5f type=client open_count=1 channel=[id: 0x9a1672cf, 0.0.0.0/0.0.0.0:45027] timeout=300.0>) 2021-05-03 05:45:24,961 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173a60 type=client open_count=1 channel=[id: 0x6513928f, 0.0.0.0/0.0.0.0:41903] timeout=300.0>) 2021-05-03 05:45:27,009 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x173a61 type=client open_count=1 channel=[id: 0xe6a3e283, 0.0.0.0/0.0.0.0:52319] timeout=300.0>) 2021-05-03 05:45:29,042 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 05:45:46,325 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 05:46:00,844 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 05:46:00,868 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #5 test_rebalance_cycles ========= 2021-05-03 05:46:00,910 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #5 test_rebalance_cycles ========= 2021-05-03 05:46:35,413 | test | INFO | pool-7-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 5.20492833794 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:46:35,440 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:46:40,539 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:46:50,759 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:46:50,803 | test | INFO | pool-7-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 15.3899998665 sec 2021-05-03 05:46:51,292 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 05:46:51,545 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 05:46:51,927 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | tgy-51-347000 | couchbase | 2 | none | 0 | 0 | 10825498624 | 0 | 0 | +---------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-03 05:46:51,928 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 05:46:55,644 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:46:55,648 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:47:25,210 | test | INFO | pool-7-thread-9 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 84.900 | ...................*..........................*..................................................**....**X | 411.808 | | 396.808 | ...............................................................................................X | 396.808 | | 358.976 | ......................................................................................X | 358.976 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:47:25,351 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 20.3491401772 | 3693387776 | 2916478976 | 0 / 3909087232 | 2201 / 4409 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 20.6428011497 | 3693387776 | 2971803648 | 0 / 3909087232 | 1849 / 3706 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 21.4102564103 | 6078930944 | 5245108224 | 0 / 3758092288 | 2301 / 4542 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 26.6701461378 | 4201836544 | 3568930816 | 0 / 3758092288 | 2307 / 4650 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 05:47:32,957 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 05:47:59,148 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | tgy-51-347000 | couchbase | 2 | none | 0 | 10000 | 10825498624 | 103669568 | 185357596 | +---------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 05:47:59,150 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #5 test_rebalance_cycles ========= 2021-05-03 05:48:06,487 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 0 2021-05-03 05:48:06,756 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:48:06,763 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:48:06,872 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:48:09,065 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:48:11,141 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:48:13,171 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:48:14,512 | test | INFO | pool-7-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 11.7023959647 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 11.4220877458 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 8.1026673377 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 9.4718131031 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:48:14,539 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:48:15,292 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:48:19,618 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:48:21,934 | test | INFO | pool-7-thread-19 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...................*...............................................**..................................*X | 453.800 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:48:24,757 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 6.93 2021-05-03 05:48:29,848 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.76 2021-05-03 05:48:34,938 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 63.54 2021-05-03 05:48:40,073 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: running, progress: 93.38 2021-05-03 05:48:50,450 | test | INFO | pool-7-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:48:50,507 | test | INFO | pool-7-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 35.996999979 sec 2021-05-03 05:48:50,549 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:48:50,555 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:48:50,584 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:48:50,694 | test | INFO | pool-7-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 42.3628908252 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 44.3068002109 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 54.7742605086 | --- OUT ---> | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 22.5974025974 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 51.5866771571 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:48:50,765 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:48:54,072 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:48:54,549 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4465 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:48:54,586 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2426 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2579 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:48:55,861 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 6.06 2021-05-03 05:48:57,601 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4548 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4554 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:48:57,940 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2621 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2700 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-2678 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2690 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-2664 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:00,210 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4807 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4764 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:00,990 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 22.4 2021-05-03 05:49:03,216 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12425 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:03,217 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12463 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12554 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12425 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:03,334 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3168 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:04,049 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5001 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:06,157 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 34.32 2021-05-03 05:49:10,161 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5137 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:10,730 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12846 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:10,732 | test | INFO | pool-7-thread-30 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12846 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:11,279 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 45.35 2021-05-03 05:49:11,664 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3516 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3553 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3486 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:12,101 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5476 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-5363 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-5411 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:13,703 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3660 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-3775 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:14,230 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5559 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5542 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:15,621 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3868 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-3830 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:15,986 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5762 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:49:15,989 | test | ERROR | pool-7-thread-25 | [task:call:3825] Failed to load 13 docs from 3920 to 5763 2021-05-03 05:49:15,990 | test | ERROR | pool-7-thread-25 | [task:call:3825] Failed to load 16 docs from 2000 to 3920 2021-05-03 05:49:15,992 | test | ERROR | pool-7-thread-25 | [task:call:3825] Failed to load 4 docs from 11600 to 13136 2021-05-03 05:49:16,420 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 56.97 2021-05-03 05:49:17,529 | test | INFO | pool-7-thread-26 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***................*......................*..............................................................*X | 448.900 | | 430.900 | ..............................................................................................X | 430.900 | | 371.700 | .................................................................................X | 371.700 | | 166.000 | ...................................*X | 166.000 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:49:21,562 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 70.79 2021-05-03 05:49:26,671 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 84.43 2021-05-03 05:49:31,802 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 99.75 2021-05-03 05:49:36,871 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:49:41,970 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:49:52,098 | test | INFO | pool-7-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x1740d1 type=client open_count=1 channel=[id: 0x6da0c22b, 0.0.0.0/0.0.0.0:43355] timeout=300.0>) 2021-05-03 05:49:52,255 | test | ERROR | pool-7-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 05:49:54,651 | test | INFO | pool-7-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 63.9570000172 sec 2021-05-03 05:49:54,653 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 1 2021-05-03 05:49:54,759 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:49:54,765 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:49:54,812 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:49:56,874 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:49:58,926 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:50:00,967 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:50:02,651 | test | INFO | pool-7-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 8.87484197219 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.57682619647 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.95021590043 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:50:02,723 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:50:03,177 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:50:07,980 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:50:13,109 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 5.55 2021-05-03 05:50:14,928 | test | INFO | pool-7-thread-3 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7131 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:16,630 | test | INFO | pool-7-thread-7 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14345 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-14437 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:16,631 | test | INFO | pool-7-thread-7 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14345 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-14437 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:18,267 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 18.05 2021-05-03 05:50:20,767 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5262 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-5173 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-5133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5158 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:21,563 | test | INFO | pool-7-thread-3 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7171 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7291 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7199 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7220 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7242 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7273 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7240 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7301 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7207 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:23,411 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.08 2021-05-03 05:50:24,040 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5350 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-5448 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5451 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5422 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-5477 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5323 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:24,789 | test | INFO | pool-7-thread-3 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7364 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7416 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7425 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:24,792 | test | ERROR | pool-7-thread-4 | [task:call:3825] Failed to load 16 docs from 5763 to 7532 2021-05-03 05:50:26,163 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5629 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5595 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5584 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:26,779 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5751 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:26,779 | test | ERROR | pool-7-thread-4 | [task:call:3825] Failed to load 14 docs from 3920 to 5763 2021-05-03 05:50:26,780 | test | ERROR | pool-7-thread-4 | [task:call:3825] Failed to load 2 docs from 13136 to 14610 2021-05-03 05:50:26,967 | test | INFO | pool-7-thread-2 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..............................*................................**......................................X | 368.300 | | 299.700 | ................................................................................X | 299.700 | | 235.700 | ..............................................................*X | 235.700 | | 153.100 | ........................................*X | 153.100 | | 147.400 | .......................................*X | 147.400 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:50:28,552 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 51.14 2021-05-03 05:50:33,714 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 71.37 2021-05-03 05:50:38,874 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: running, progress: 92.66 2021-05-03 05:50:49,076 | test | INFO | pool-7-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:50:49,111 | test | INFO | pool-7-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 46.4599997997 sec 2021-05-03 05:50:49,233 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:50:49,237 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:50:49,381 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:50:49,523 | test | INFO | pool-7-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 86.9711042312 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 44.3388756928 | Cluster node | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 19.9224806202 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 46.9084268197 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:50:49,578 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:50:52,621 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:50:54,674 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 7.7 2021-05-03 05:50:58,723 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8086 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7947 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8116 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:59,181 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6183 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6270 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6360 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6235 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6342 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6217 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:59,645 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15191 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15079 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:59,645 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15191 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15079 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:50:59,938 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 24.3 2021-05-03 05:51:03,825 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8140 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8155 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8226 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:03,881 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6539 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6366 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6414 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6457 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6434 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:05,029 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 35.98 2021-05-03 05:51:10,127 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 50.9 2021-05-03 05:51:11,178 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8452 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8496 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8361 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8388 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8456 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8338 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8413 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:11,266 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6681 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6605 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6719 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6674 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6595 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6689 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6711 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6622 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:11,940 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15440 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15462 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15595 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15537 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-15607 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:11,941 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15440 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15462 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15595 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15537 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15607 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:15,234 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.96 2021-05-03 05:51:16,792 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6891 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6901 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6806 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:16,887 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:16,888 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15667 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:16,973 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8667 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:20,316 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 81.84 2021-05-03 05:51:20,994 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7075 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6971 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:21,029 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8790 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8928 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:23,743 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:24,788 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9077 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9102 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-9092 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8973 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9018 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8996 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:25,480 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 96.72 2021-05-03 05:51:26,187 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7370 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7441 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-7439 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7509 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7499 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7447 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7402 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:26,739 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9145 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9225 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:51:26,742 | test | ERROR | pool-7-thread-9 | [task:call:3825] Failed to load 30 docs from 7532 to 9230 2021-05-03 05:51:26,743 | test | ERROR | pool-7-thread-9 | [task:call:3825] Failed to load 32 docs from 5763 to 7532 2021-05-03 05:51:26,743 | test | ERROR | pool-7-thread-9 | [task:call:3825] Failed to load 8 docs from 14610 to 16025 2021-05-03 05:51:28,171 | test | INFO | pool-7-thread-8 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **............................................*.........................**...............................X | 296.300 | | 227.000 | ...........................................................................X | 227.000 | | 177.300 | ..........................................................*X | 177.300 | | 116.700 | ......................................X | 116.700 | | 85.900 | ...........................*.....**.*........*X | 126.900 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:51:30,611 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:51:35,713 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:51:45,832 | test | INFO | pool-7-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:51:46,002 | test | INFO | pool-7-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 56.4790000916 sec 2021-05-03 05:51:46,003 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 2 2021-05-03 05:51:46,062 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:51:46,071 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:51:46,121 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:51:48,148 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:51:50,181 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:51:52,239 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:51:53,930 | test | INFO | pool-7-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 4.64411913175 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 22 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.12655338575 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:51:53,977 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:51:54,374 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:51:59,095 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:52:04,233 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 4.26 2021-05-03 05:52:04,788 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10414 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:05,217 | test | INFO | pool-7-thread-19 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:05,219 | test | INFO | pool-7-thread-19 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:06,852 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8561 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8552 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8680 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8570 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8719 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8579 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8689 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8535 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8610 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8710 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:09,706 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 17.5 2021-05-03 05:52:12,019 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10469 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10436 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10600 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10473 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10569 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10471 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:14,802 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 35.54 2021-05-03 05:52:14,878 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8818 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8825 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8803 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8746 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8762 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:15,316 | test | INFO | pool-7-thread-19 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-17028 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:15,318 | test | INFO | pool-7-thread-19 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-17028 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:16,354 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10661 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10723 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10779 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10768 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10725 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:17,464 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9049 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9127 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:18,500 | test | INFO | pool-7-thread-22 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10851 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-10853 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:18,500 | test | ERROR | pool-7-thread-16 | [task:call:3825] Failed to load 18 docs from 9230 to 10860 2021-05-03 05:52:19,306 | test | INFO | pool-7-thread-21 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9181 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:19,308 | test | ERROR | pool-7-thread-16 | [task:call:3825] Failed to load 20 docs from 7532 to 9230 2021-05-03 05:52:19,309 | test | ERROR | pool-7-thread-16 | [task:call:3825] Failed to load 2 docs from 16025 to 17384 2021-05-03 05:52:19,938 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 54.45 2021-05-03 05:52:20,239 | test | INFO | pool-7-thread-20 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...............................*..............................**.......................................*X | 306.200 | | 264.200 | .....................................................................................X | 264.200 | | 216.700 | .....................................................................*X | 216.700 | | 145.300 | ..............................................X | 145.300 | | 128.900 | .........................................*..*X | 135.500 | | 113.900 | ....................................X | 113.900 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:52:25,048 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 76.4 2021-05-03 05:52:30,147 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: running, progress: 99.18 2021-05-03 05:52:40,273 | test | INFO | pool-7-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:52:40,311 | test | INFO | pool-7-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 46.379999876 sec 2021-05-03 05:52:40,365 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:52:40,375 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:52:40,415 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:52:40,509 | test | INFO | pool-7-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 35.9059545106 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 42.3187647213 | Cluster node | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 20.8171709335 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 62.1113143454 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:52:40,542 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:52:43,486 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:52:44,743 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9738 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:44,921 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11291 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11384 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11301 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:45,668 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 6.37 2021-05-03 05:52:49,996 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9848 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9954 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-9923 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9997 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:50,776 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11650 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11573 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11520 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11523 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11548 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11606 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11628 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:50,809 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 23.44 2021-05-03 05:52:55,749 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10045 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:55,828 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11783 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11737 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:55,971 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 34.57 2021-05-03 05:52:57,805 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:52:57,805 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18159 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:00,990 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10339 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10416 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10244 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10321 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10387 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10364 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:01,148 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 50.47 2021-05-03 05:53:01,180 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12047 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11894 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12015 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11999 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11956 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11904 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:04,595 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10550 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10466 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:05,030 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12072 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-12196 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-12186 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12239 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:06,365 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 65.28 2021-05-03 05:53:08,362 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18648 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:08,364 | test | INFO | pool-7-thread-26 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18648 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:08,450 | test | INFO | pool-7-thread-29 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12381 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12288 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-12399 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12318 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:08,451 | test | ERROR | pool-7-thread-17 | [task:call:3825] Failed to load 27 docs from 10860 to 12425 2021-05-03 05:53:08,451 | test | INFO | pool-7-thread-28 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10692 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10702 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10816 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:08,973 | test | ERROR | pool-7-thread-17 | [task:call:3825] Failed to load 18 docs from 9230 to 10860 2021-05-03 05:53:08,974 | test | ERROR | pool-7-thread-17 | [task:call:3825] Failed to load 2 docs from 17384 to 18688 2021-05-03 05:53:10,013 | test | INFO | pool-7-thread-27 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..................................**...............................*...................................X | 284.780 | | 284.180 | ..................................................................................................*X | 284.180 | | 224.080 | .............................................................................X | 224.080 | | 152.005 | ....................................................X | 152.005 | | 86.705 | .............................**...X | 96.405 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:53:11,529 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 80.08 2021-05-03 05:53:16,680 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 96.25 2021-05-03 05:53:21,839 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:53:27,000 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:53:37,221 | test | INFO | pool-7-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:53:37,615 | test | INFO | pool-7-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 57.1059999466 sec 2021-05-03 05:53:37,617 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:941] Cycle 3 2021-05-03 05:53:37,661 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:53:37,667 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:53:37,713 | test | INFO | pool-7-thread-7 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12425 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:37,719 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:53:37,746 | test | INFO | pool-7-thread-7 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12846 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:37,868 | test | ERROR | pool-7-thread-24 | [task:call:3825] Failed to load 4 docs from 12425 to 13928 2021-05-03 05:53:39,759 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:53:41,819 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:53:43,898 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:53:45,371 | test | INFO | pool-7-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.83997980818 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 5.90909090909 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.81527531083 | Cluster node | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:53:45,440 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:53:46,226 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:53:50,589 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:53:55,752 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 3.71 2021-05-03 05:53:56,125 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12079 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:57,882 | test | INFO | pool-7-thread-6 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:53:57,884 | test | ERROR | pool-7-thread-24 | [task:call:3825] Failed to load 3 docs from 10860 to 12425 2021-05-03 05:53:58,000 | test | INFO | pool-7-thread-1 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..........................*.............................**...........................................**..X | 323.400 | | 307.000 | .............................................................................................X | 307.000 | | 209.700 | ...............................................................X | 209.700 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:54:00,838 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 18.36 2021-05-03 05:54:05,961 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 37.11 2021-05-03 05:54:11,104 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 58.64 2021-05-03 05:54:16,249 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: running, progress: 81.81 2021-05-03 05:54:26,424 | test | INFO | pool-7-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:54:26,470 | test | INFO | pool-7-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 41.0999999046 sec 2021-05-03 05:54:26,528 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:54:26,536 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:54:26,576 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-03 05:54:26,630 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14345 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-14437 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:26,665 | test | INFO | pool-7-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 82.1345105167 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 39.7435897436 | Cluster node | | 172.23.120.206 | kv | 7.0.0-5085-enterprise | 21.0349288486 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 47.5371383894 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:54:26,677 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15079 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:26,694 | test | INFO | pool-7-thread-14 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15191 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:26,698 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:54:26,710 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 4 docs from 13928 to 15370 2021-05-03 05:54:29,124 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14166 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:29,398 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-03 05:54:31,813 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 7.15 2021-05-03 05:54:35,996 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12880 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-13001 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-13014 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12872 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12895 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12940 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12910 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12856 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12989 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12905 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:36,015 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20359 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:36,016 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20359 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:36,121 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14385 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:36,930 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 24.65 2021-05-03 05:54:40,825 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14551 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14595 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14641 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-14555 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:41,052 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13141 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-13221 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13048 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13047 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:41,125 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20550 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:41,127 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20550 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:42,086 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 36.64 2021-05-03 05:54:47,270 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 51.06 2021-05-03 05:54:47,829 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13260 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13392 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13270 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13251 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13378 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13225 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13249 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:47,898 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14860 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-14761 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14743 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14789 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-14888 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14844 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14811 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14918 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:48,020 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:48,022 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:52,385 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 66.53 2021-05-03 05:54:52,404 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13480 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13510 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13555 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13569 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:52,476 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15062 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15087 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15117 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14983 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14966 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-14969 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:52,500 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-21132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-21096 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-21106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:52,500 | test | INFO | pool-7-thread-8 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-21132 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-21096 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-21106 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:56,427 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13677 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:56,499 | test | INFO | pool-7-thread-13 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15170 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15195 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15223 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15289 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15235 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-15319 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:57,400 | test | INFO | pool-7-thread-12 | [table_view:display:72] Keys failed in tgy-51-347000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13843 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 05:54:57,487 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 81.17 2021-05-03 05:54:57,642 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 28 docs from 13928 to 15370 2021-05-03 05:54:57,644 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 29 docs from 12425 to 13928 2021-05-03 05:54:57,644 | test | ERROR | pool-7-thread-5 | [task:call:3825] Failed to load 7 docs from 19940 to 21142 2021-05-03 05:54:57,776 | test | INFO | pool-7-thread-11 | [table_view:display:72] Ops trend for bucket 'tgy-51-347000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **......................*.................................................**.............................X | 296.947 | | 262.768 | .......................................................................................X | 262.768 | | 152.768 | ..................................................*X | 152.768 | | 104.468 | ..................................**...*X | 112.861 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:55:02,586 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 98.94 2021-05-03 05:55:07,727 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:55:12,898 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 05:55:23,007 | test | INFO | pool-7-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:55:23,203 | test | INFO | pool-7-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 56.5380001068 sec 2021-05-03 05:55:24,484 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: tgy-51-347000, Active Resident ratio(DGM): 100% 2021-05-03 05:55:24,486 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: tgy-51-347000, Replica Resident ratio(DGM): 100% 2021-05-03 05:55:47,032 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 05:56:01,130 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.4080002308 seconds to complete 2021-05-03 05:56:01,148 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 763.613s OK During the test, Remote Connections: 62, Disconnections: 62 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_5 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,graceful=False' Test Input params: {'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'graceful': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_6', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 6, 'bulk_api_crud': 'True'} test_Orchestrator_Node_failover (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 05:56:12,417 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 05:56:13,674 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 05:56:14,780 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x175348 type=client open_count=1 channel=[id: 0xdf8b0d80, 0.0.0.0/0.0.0.0:45233] timeout=300.0>) 2021-05-03 05:56:21,190 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x175349 type=client open_count=1 channel=[id: 0xc4a5c647, 0.0.0.0/0.0.0.0:35057] timeout=300.0>) 2021-05-03 05:56:23,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17534a type=client open_count=1 channel=[id: 0x3b92d2cb, 0.0.0.0/0.0.0.0:39943] timeout=300.0>) 2021-05-03 05:56:25,328 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17534b type=client open_count=1 channel=[id: 0xe1e7d523, 0.0.0.0/0.0.0.0:43525] timeout=300.0>) 2021-05-03 05:56:27,375 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17534c type=client open_count=1 channel=[id: 0xbabb1b44, 0.0.0.0/0.0.0.0:59621] timeout=300.0>) 2021-05-03 05:56:30,430 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17534d type=client open_count=1 channel=[id: 0xacc99ff6, 0.0.0.0/0.0.0.0:39549] timeout=300.0>) 2021-05-03 05:56:32,496 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17534e type=client open_count=1 channel=[id: 0x63f2f2fc, 0.0.0.0/0.0.0.0:45521] timeout=300.0>) 2021-05-03 05:56:34,519 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x17534f type=client open_count=1 channel=[id: 0x155931a9, 0.0.0.0/0.0.0.0:53229] timeout=300.0>) 2021-05-03 05:56:36,552 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175350 type=client open_count=1 channel=[id: 0xec68992b, 0.0.0.0/0.0.0.0:53797] timeout=300.0>) 2021-05-03 05:56:45,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175351 type=client open_count=1 channel=[id: 0xd03d62e8, 0.0.0.0/0.0.0.0:60349] timeout=300.0>) 2021-05-03 05:56:47,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175352 type=client open_count=1 channel=[id: 0xa0b5e2b9, 0.0.0.0/0.0.0.0:54085] timeout=300.0>) 2021-05-03 05:56:49,688 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175353 type=client open_count=1 channel=[id: 0xc1d54560, 0.0.0.0/0.0.0.0:34054] timeout=300.0>) 2021-05-03 05:56:51,720 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175354 type=client open_count=1 channel=[id: 0x0d6ddea9, 0.0.0.0/0.0.0.0:59817] timeout=300.0>) 2021-05-03 05:56:53,733 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175355 type=client open_count=1 channel=[id: 0xed64f7f4, 0.0.0.0/0.0.0.0:47180] timeout=300.0>) 2021-05-03 05:56:55,756 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175356 type=client open_count=1 channel=[id: 0x4b4fdc7f, 0.0.0.0/0.0.0.0:50549] timeout=300.0>) 2021-05-03 05:56:57,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175357 type=client open_count=1 channel=[id: 0x5a74c3d8, 0.0.0.0/0.0.0.0:39883] timeout=300.0>) 2021-05-03 05:56:59,832 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x175358 type=client open_count=1 channel=[id: 0xfecf7521, 0.0.0.0/0.0.0.0:49245] timeout=300.0>) 2021-05-03 05:57:01,868 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x175359 type=client open_count=1 channel=[id: 0xa39ffb0b, 0.0.0.0/0.0.0.0:45729] timeout=300.0>) 2021-05-03 05:57:10,239 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17535a type=client open_count=1 channel=[id: 0x9581862e, 0.0.0.0/0.0.0.0:59815] timeout=300.0>) 2021-05-03 05:57:12,278 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17535b type=client open_count=1 channel=[id: 0x32ddb6fa, 0.0.0.0/0.0.0.0:44567] timeout=300.0>) 2021-05-03 05:57:14,303 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17535c type=client open_count=1 channel=[id: 0x6f0582f6, 0.0.0.0/0.0.0.0:49613] timeout=300.0>) 2021-05-03 05:57:16,332 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17535d type=client open_count=1 channel=[id: 0x4c0b41e5, 0.0.0.0/0.0.0.0:60785] timeout=300.0>) 2021-05-03 05:57:18,378 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17535e type=client open_count=1 channel=[id: 0x24268eda, 0.0.0.0/0.0.0.0:42035] timeout=300.0>) 2021-05-03 05:57:20,391 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17535f type=client open_count=1 channel=[id: 0x58722973, 0.0.0.0/0.0.0.0:43659] timeout=300.0>) 2021-05-03 05:57:22,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x175360 type=client open_count=1 channel=[id: 0x9bf5d538, 0.0.0.0/0.0.0.0:56455] timeout=300.0>) 2021-05-03 05:57:24,437 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x175361 type=client open_count=1 channel=[id: 0x2f286b2c, 0.0.0.0/0.0.0.0:53669] timeout=300.0>) 2021-05-03 05:57:26,460 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x175362 type=client open_count=1 channel=[id: 0xfb151217, 0.0.0.0/0.0.0.0:42511] timeout=300.0>) 2021-05-03 05:57:28,483 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175363 type=client open_count=1 channel=[id: 0xbf564f32, 0.0.0.0/0.0.0.0:50499] timeout=300.0>) 2021-05-03 05:57:36,525 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175364 type=client open_count=1 channel=[id: 0x59b50c73, 0.0.0.0/0.0.0.0:58895] timeout=300.0>) 2021-05-03 05:57:38,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175365 type=client open_count=1 channel=[id: 0x82e9c40d, 0.0.0.0/0.0.0.0:39595] timeout=300.0>) 2021-05-03 05:57:40,630 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175366 type=client open_count=1 channel=[id: 0x568cbc17, 0.0.0.0/0.0.0.0:54860] timeout=300.0>) 2021-05-03 05:57:42,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175367 type=client open_count=1 channel=[id: 0x53ec8974, 0.0.0.0/0.0.0.0:49071] timeout=300.0>) 2021-05-03 05:57:44,670 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175368 type=client open_count=1 channel=[id: 0x245385a7, 0.0.0.0/0.0.0.0:57602] timeout=300.0>) 2021-05-03 05:57:46,698 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175369 type=client open_count=1 channel=[id: 0x08bb7bb4, 0.0.0.0/0.0.0.0:42619] timeout=300.0>) 2021-05-03 05:57:48,730 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17536a type=client open_count=1 channel=[id: 0x393c41ad, 0.0.0.0/0.0.0.0:37186] timeout=300.0>) 2021-05-03 05:57:50,752 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17536b type=client open_count=1 channel=[id: 0xedc450dd, 0.0.0.0/0.0.0.0:49982] timeout=300.0>) 2021-05-03 05:57:52,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17536c type=client open_count=1 channel=[id: 0xc1e01b0c, 0.0.0.0/0.0.0.0:35059] timeout=300.0>) 2021-05-03 05:57:57,806 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17536d type=client open_count=1 channel=[id: 0x8083831e, 0.0.0.0/0.0.0.0:54081] timeout=300.0>) 2021-05-03 05:57:59,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17536e type=client open_count=1 channel=[id: 0x1ba0f9a3, 0.0.0.0/0.0.0.0:34836] timeout=300.0>) 2021-05-03 05:58:01,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x17536f type=client open_count=1 channel=[id: 0x1aef8ff7, 0.0.0.0/0.0.0.0:57205] timeout=300.0>) 2021-05-03 05:58:03,898 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x175370 type=client open_count=1 channel=[id: 0x5918ed43, 0.0.0.0/0.0.0.0:33714] timeout=300.0>) 2021-05-03 05:58:05,924 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x175371 type=client open_count=1 channel=[id: 0x30e9ae8a, 0.0.0.0/0.0.0.0:39343] timeout=300.0>) 2021-05-03 05:58:07,951 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 05:58:26,322 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 05:58:41,338 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 05:58:41,408 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 05:58:41,614 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 05:59:13,743 | test | INFO | pool-8-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 4.10372608258 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 05:59:13,852 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 05:59:24,042 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 05:59:24,073 | test | INFO | pool-8-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.3299999237 sec 2021-05-03 05:59:24,627 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 05:59:25,319 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 05:59:25,505 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | o3l6BBv9-24-775000 | couchbase | 2 | none | 0 | 0 | 10825498624 | 0 | 0 | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-03 05:59:25,506 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 05:59:28,355 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 05:59:28,361 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 05:59:58,371 | test | INFO | pool-8-thread-9 | [table_view:display:72] Ops trend for bucket 'o3l6BBv9-24-775000' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 90.827 | ......................*...................................................**..........................*X | 394.327 | | 393.500 | ..................................................................................................*X | 393.500 | | 372.100 | .............................................................................................*X | 372.100 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 05:59:58,434 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 22.1843003413 | 3693387776 | 2903060480 | 0 / 3909087232 | 1851 / 3709 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 20.4646306447 | 3693387776 | 2965786624 | 0 / 3909087232 | 2249 / 4557 | 7.0.0-5085-enterprise | | 172.23.104.186 | kv | 19.3335029255 | 6078930944 | 5246812160 | 0 / 3758092288 | 1979 / 3919 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 25.5313634007 | 4201836544 | 3576332288 | 0 / 3758092288 | 1979 / 3955 | 7.0.0-5085-enterprise | +----------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 06:00:04,848 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 06:00:28,161 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | o3l6BBv9-24-775000 | couchbase | 2 | none | 0 | 10000 | 10825498624 | 103776528 | 179000784 | +--------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 06:00:28,163 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #6 test_Orchestrator_Node_failover ========= 2021-05-03 06:00:41,605 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 06:01:05,250 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 0 2021-05-03 06:01:15,226 | test | INFO | pool-8-thread-23 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.24300003052 seconds to complete 2021-05-03 06:01:15,232 | test | INFO | pool-8-thread-23 | [common_lib:sleep:22] Sleep 8.24300003052 seconds. Reason: Wait after rebalance complete 2021-05-03 06:01:23,717 | test | INFO | pool-8-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 3.26499620349 | --- OUT ---> | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 8.28687278256 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 11.9748427673 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 10.4515474378 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:01:23,736 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:01:28,838 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 14.32 2021-05-03 06:01:33,994 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 37.63 2021-05-03 06:01:39,157 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 61.07 2021-05-03 06:01:44,290 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: running, progress: 85.3 Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x175472 type=client open_count=1 channel=[id: 0xe151c45b, 0.0.0.0/0.0.0.0:37389] timeout=300.0>) 2021-05-03 06:01:49,551 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x175473 type=client open_count=1 channel=[id: 0x2a7b44ac, 0.0.0.0/0.0.0.0:59153] timeout=300.0>) 2021-05-03 06:01:51,559 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x175474 type=client open_count=1 channel=[id: 0xc64fcfc6, 0.0.0.0/0.0.0.0:46571] timeout=300.0>) 2021-05-03 06:01:53,591 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x175475 type=client open_count=1 channel=[id: 0x98dbc58b, 0.0.0.0/0.0.0.0:52073] timeout=300.0>) 2021-05-03 06:01:55,601 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x175476 type=client open_count=1 channel=[id: 0x5f701839, 0.0.0.0/0.0.0.0:50039] timeout=300.0>) 2021-05-03 06:01:57,621 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/pools/default/tasks error [Errno 111] Connection refused 2021-05-03 06:02:04,770 | test | INFO | pool-8-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:02:04,796 | test | ERROR | pool-8-thread-24 | [rest_client:_http_request:748] GET http://172.23.98.196:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.98.196:8091/pools/default with status 0: unknown pool 2021-05-03 06:02:04,895 | test | INFO | pool-8-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 41.1779999733 sec 2021-05-03 06:02:05,111 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.104.186 2021-05-03 06:02:05,111 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.98.196 2021-05-03 06:02:17,862 | test | INFO | pool-8-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.55611601513 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 4.57171564933 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 4.59828196059 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:02:17,937 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:02:23,023 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:02:28,118 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 18.94 2021-05-03 06:02:33,305 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 41.33 2021-05-03 06:02:38,441 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 62.34 2021-05-03 06:02:43,569 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: running, progress: 83.71 2021-05-03 06:02:53,772 | test | INFO | pool-8-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:02:53,808 | test | INFO | pool-8-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 35.9460000992 sec 2021-05-03 06:02:53,811 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 1 2021-05-03 06:03:03,769 | test | INFO | pool-8-thread-26 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.15700006485 seconds to complete 2021-05-03 06:03:03,775 | test | INFO | pool-8-thread-26 | [common_lib:sleep:22] Sleep 8.15700006485 seconds. Reason: Wait after rebalance complete 2021-05-03 06:03:12,128 | test | INFO | pool-8-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 2.40263024785 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 7.78990104035 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 8.46560846561 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 10.7052257737 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:03:12,164 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:03:17,270 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 16.64 2021-05-03 06:03:22,352 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 40.73 2021-05-03 06:03:27,446 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 62.63 2021-05-03 06:03:32,572 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: running, progress: 87.6 2021-05-03 06:03:42,709 | test | INFO | pool-8-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175477 type=client open_count=1 channel=[id: 0xe7ca2552, 0.0.0.0/0.0.0.0:54661] timeout=300.0>) 2021-05-03 06:03:42,750 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175478 type=client open_count=1 channel=[id: 0x16e8eb7b, 0.0.0.0/0.0.0.0:46399] timeout=300.0>) 2021-05-03 06:03:44,763 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x175479 type=client open_count=1 channel=[id: 0xe2df8ae3, 0.0.0.0/0.0.0.0:34327] timeout=300.0>) 2021-05-03 06:03:46,776 | test | ERROR | pool-8-thread-27 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 06:03:48,839 | test | INFO | pool-8-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 36.7119998932 sec 2021-05-03 06:03:48,936 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.121.10 2021-05-03 06:03:48,937 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.104.186 2021-05-03 06:04:00,259 | test | INFO | pool-8-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 4.30488731324 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 6.37490513534 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 7.80894617134 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:04:00,335 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:04:05,423 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:04:10,542 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:04:15,634 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 14.02 2021-05-03 06:04:20,746 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 32.88 2021-05-03 06:04:25,875 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 52.18 2021-05-03 06:04:31,003 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.11 2021-05-03 06:04:36,157 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: running, progress: 91.36 2021-05-03 06:04:46,390 | test | INFO | pool-8-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:04:46,427 | test | INFO | pool-8-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 46.1690001488 sec 2021-05-03 06:04:46,428 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 2 2021-05-03 06:04:56,361 | test | INFO | pool-8-thread-29 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.21600008011 seconds to complete 2021-05-03 06:04:56,367 | test | INFO | pool-8-thread-29 | [common_lib:sleep:22] Sleep 8.21600008011 seconds. Reason: Wait after rebalance complete 2021-05-03 06:05:04,864 | test | INFO | pool-8-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 7.8431372549 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 9.09783989835 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 3.58306188925 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 9.63000506842 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:05:04,918 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:05:10,052 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 13.61 2021-05-03 06:05:15,183 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 34.55 2021-05-03 06:05:20,367 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 54.96 2021-05-03 06:05:25,532 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 82.63 2021-05-03 06:05:30,698 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 06:05:40,851 | test | INFO | pool-8-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x17547a type=client open_count=1 channel=[id: 0x6fbba312, 0.0.0.0/0.0.0.0:37687] timeout=300.0>) 2021-05-03 06:05:40,938 | test | ERROR | pool-8-thread-30 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 06:05:43,226 | test | INFO | pool-8-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 38.364000082 sec 2021-05-03 06:05:43,336 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.104.186 2021-05-03 06:05:43,336 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.121.10 2021-05-03 06:05:55,361 | test | INFO | pool-8-thread-2 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 6.11675126904 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 3.83838383838 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 4.86337427927 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:05:55,403 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:06:00,576 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:06:05,701 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:06:10,874 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 9.08 2021-05-03 06:06:15,979 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 32.99 2021-05-03 06:06:21,073 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 51.92 2021-05-03 06:06:26,153 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 73.05 2021-05-03 06:06:31,242 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: running, progress: 90.62 2021-05-03 06:06:41,493 | test | INFO | pool-8-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:06:41,532 | test | INFO | pool-8-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 46.1699998379 sec 2021-05-03 06:06:41,532 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:977] Cycle 3 2021-05-03 06:06:51,082 | test | INFO | pool-8-thread-1 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.1890001297 seconds to complete 2021-05-03 06:06:51,091 | test | INFO | pool-8-thread-1 | [common_lib:sleep:22] Sleep 8.1890001297 seconds. Reason: Wait after rebalance complete 2021-05-03 06:06:59,500 | test | INFO | pool-8-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 2.27445034117 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 2.79808419461 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 7.89143000754 | --- OUT ---> | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 3.41426403642 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:06:59,542 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:07:04,700 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 19.5 2021-05-03 06:07:09,808 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 48.55 2021-05-03 06:07:14,960 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 74.62 2021-05-03 06:07:20,198 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: running, progress: 99.52 2021-05-03 06:07:30,319 | test | INFO | pool-8-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17547b type=client open_count=1 channel=[id: 0xcc434bf6, 0.0.0.0/0.0.0.0:53576] timeout=300.0>) 2021-05-03 06:07:30,460 | test | ERROR | pool-8-thread-5 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x17547c type=client open_count=1 channel=[id: 0xc0e9ea7f, 0.0.0.0/0.0.0.0:43397] timeout=300.0>) 2021-05-03 06:07:32,500 | test | ERROR | pool-8-thread-5 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 06:07:34,621 | test | INFO | pool-8-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 35.1210000515 sec 2021-05-03 06:07:34,845 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1017] master node is:172.23.121.10 2021-05-03 06:07:34,845 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1018] Added node is:172.23.104.186 2021-05-03 06:07:45,641 | test | INFO | pool-8-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 5.89131538852 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 4.80161738691 | Cluster node | | 172.23.104.186 | kv | 7.0.0-5085-enterprise | 0 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 6.40182186235 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:07:45,806 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:07:50,940 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:07:56,030 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:08:01,153 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 11.99 2021-05-03 06:08:06,365 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 30.43 2021-05-03 06:08:11,479 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 49.72 2021-05-03 06:08:16,579 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.39 2021-05-03 06:08:21,716 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: running, progress: 92.41 2021-05-03 06:08:31,864 | test | INFO | pool-8-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:08:31,905 | test | INFO | pool-8-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 46.265999794 sec 2021-05-03 06:08:33,476 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: o3l6BBv9-24-775000, Active Resident ratio(DGM): 100% 2021-05-03 06:08:33,476 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: o3l6BBv9-24-775000, Replica Resident ratio(DGM): 100% 2021-05-03 06:08:55,391 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 06:09:09,703 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2050001621 seconds to complete 2021-05-03 06:09:09,726 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 788.662s OK During the test, Remote Connections: 82, Disconnections: 82 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 6 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_03-07-02/test_6 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=3,nodes_swap=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=3,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' skipped, GROUP not satisfied Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 3h 2m 29s 2 actionable tasks: 1 executed, 1 up-to-date + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even/build/classes/java/main:/data/workspace/temp_rebalance_even/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_rebalance_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'cluster_name': 'win10-bucket-ops', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops.ini', 'num_nodes': 5, 'quota_percent': '95', 'rerun': 'False', 'spec': 'collections_rebalance_crud_on_collections'} Only cases in GROUPs 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' will be executed Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_06-09-34/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_06-09-34/test_1', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 1, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 06:09:45,588 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 06:09:46,941 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 06:09:49,684 | test | ERROR | MainThread | [rest_client:_http_request:748] GET http://172.23.98.196:8091/pools/default/ body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-05-03 06:09:50,328 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0x193ac56f, 0.0.0.0/0.0.0.0:38967] timeout=300.0>) 2021-05-03 06:09:56,584 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x7ed86113, 0.0.0.0/0.0.0.0:47645] timeout=300.0>) 2021-05-03 06:10:00,338 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x32964b8d, 0.0.0.0/0.0.0.0:40867] timeout=300.0>) 2021-05-03 06:10:02,457 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x7d1a8562, 0.0.0.0/0.0.0.0:41255] timeout=300.0>) 2021-05-03 06:10:04,500 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xc527f755, 0.0.0.0/0.0.0.0:59406] timeout=300.0>) 2021-05-03 06:10:06,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xead49736, 0.0.0.0/0.0.0.0:35512] timeout=300.0>) 2021-05-03 06:10:08,628 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x1e6ad15b, 0.0.0.0/0.0.0.0:54829] timeout=300.0>) 2021-05-03 06:10:10,668 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x1198621e, 0.0.0.0/0.0.0.0:52560] timeout=300.0>) 2021-05-03 06:10:12,713 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x90207d0c, 0.0.0.0/0.0.0.0:50376] timeout=300.0>) 2021-05-03 06:10:14,796 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0xd7079234, 0.0.0.0/0.0.0.0:58237] timeout=300.0>) 2021-05-03 06:10:24,257 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0xd8cbfc81, 0.0.0.0/0.0.0.0:37415] timeout=300.0>) 2021-05-03 06:10:26,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x76f129db, 0.0.0.0/0.0.0.0:51871] timeout=300.0>) 2021-05-03 06:10:28,401 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0xa6c31aaa, 0.0.0.0/0.0.0.0:53173] timeout=300.0>) 2021-05-03 06:10:30,448 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x817a788b, 0.0.0.0/0.0.0.0:58403] timeout=300.0>) 2021-05-03 06:10:32,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x7acaf6f0, 0.0.0.0/0.0.0.0:35967] timeout=300.0>) 2021-05-03 06:10:34,506 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x438afcc4, 0.0.0.0/0.0.0.0:45037] timeout=300.0>) 2021-05-03 06:10:36,592 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x99f4f015, 0.0.0.0/0.0.0.0:43251] timeout=300.0>) 2021-05-03 06:10:38,622 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0x5f9faad9, 0.0.0.0/0.0.0.0:55149] timeout=300.0>) 2021-05-03 06:10:40,647 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0xe60538eb, 0.0.0.0/0.0.0.0:46005] timeout=300.0>) 2021-05-03 06:10:49,885 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x7547d18e, 0.0.0.0/0.0.0.0:56827] timeout=300.0>) 2021-05-03 06:10:51,928 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0x7bf2f7bd, 0.0.0.0/0.0.0.0:37814] timeout=300.0>) 2021-05-03 06:10:54,002 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x9c9889a7, 0.0.0.0/0.0.0.0:41455] timeout=300.0>) 2021-05-03 06:10:56,049 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0x6b93f653, 0.0.0.0/0.0.0.0:51855] timeout=300.0>) 2021-05-03 06:10:58,167 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x174af49f, 0.0.0.0/0.0.0.0:38777] timeout=300.0>) 2021-05-03 06:11:00,250 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0x029a98bf, 0.0.0.0/0.0.0.0:33171] timeout=300.0>) 2021-05-03 06:11:02,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0x938836ac, 0.0.0.0/0.0.0.0:35285] timeout=300.0>) 2021-05-03 06:11:04,348 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0xc3226a20, 0.0.0.0/0.0.0.0:58573] timeout=300.0>) 2021-05-03 06:11:06,451 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0xd5f93104, 0.0.0.0/0.0.0.0:39802] timeout=300.0>) 2021-05-03 06:11:08,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0x4db01433, 0.0.0.0/0.0.0.0:49059] timeout=300.0>) 2021-05-03 06:11:18,069 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2a type=client open_count=1 channel=[id: 0x40a73c41, 0.0.0.0/0.0.0.0:45129] timeout=300.0>) 2021-05-03 06:11:20,131 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2b type=client open_count=1 channel=[id: 0xd0ca57c1, 0.0.0.0/0.0.0.0:51581] timeout=300.0>) 2021-05-03 06:11:22,203 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2c type=client open_count=1 channel=[id: 0xb33aa741, 0.0.0.0/0.0.0.0:54180] timeout=300.0>) 2021-05-03 06:11:24,301 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2d type=client open_count=1 channel=[id: 0xee593832, 0.0.0.0/0.0.0.0:38435] timeout=300.0>) 2021-05-03 06:11:26,348 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2e type=client open_count=1 channel=[id: 0xb7cc67df, 0.0.0.0/0.0.0.0:49171] timeout=300.0>) 2021-05-03 06:11:28,423 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x2f type=client open_count=1 channel=[id: 0xf53c4739, 0.0.0.0/0.0.0.0:41863] timeout=300.0>) 2021-05-03 06:11:30,513 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x30 type=client open_count=1 channel=[id: 0x8439b1ae, 0.0.0.0/0.0.0.0:56499] timeout=300.0>) 2021-05-03 06:11:32,602 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x31 type=client open_count=1 channel=[id: 0xb206d960, 0.0.0.0/0.0.0.0:47351] timeout=300.0>) 2021-05-03 06:11:34,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x32 type=client open_count=1 channel=[id: 0xacfd86f4, 0.0.0.0/0.0.0.0:58977] timeout=300.0>) 2021-05-03 06:11:39,937 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x33 type=client open_count=1 channel=[id: 0x1931707f, 0.0.0.0/0.0.0.0:55483] timeout=300.0>) 2021-05-03 06:11:41,973 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x34 type=client open_count=1 channel=[id: 0xb3268d6d, 0.0.0.0/0.0.0.0:42843] timeout=300.0>) 2021-05-03 06:11:44,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x35 type=client open_count=1 channel=[id: 0x526f2611, 0.0.0.0/0.0.0.0:50084] timeout=300.0>) 2021-05-03 06:11:46,053 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x36 type=client open_count=1 channel=[id: 0xacaf256f, 0.0.0.0/0.0.0.0:57859] timeout=300.0>) 2021-05-03 06:11:48,088 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x37 type=client open_count=1 channel=[id: 0x73628147, 0.0.0.0/0.0.0.0:42153] timeout=300.0>) 2021-05-03 06:11:50,125 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 06:12:08,032 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 06:12:23,069 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 06:12:23,104 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 06:12:23,234 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 06:12:45,334 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 7.10148577185 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:12:45,410 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:12:55,802 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:12:55,851 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.5320000648 sec 2021-05-03 06:12:56,338 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 06:13:45,947 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 06:13:46,322 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 629145600 | 687548952 | 42508976 | | bucket2 | couchbase | 2 | none | 0 | 0 | 629145600 | 667427224 | 65872692 | | default | couchbase | 2 | none | 0 | 0 | 6291456000 | 917710200 | 27082389 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-03 06:13:46,325 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 06:15:05,673 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 06:15:05,759 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 06:36:40,295 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****.........*................*X | 16.100 | | 10.600 | ................X | 10.600 | | 5.100 | .......X | 5.100 | | 0.000 | **********************************************************************....*.........*...................**.......................*...............*X | 43.397 | | 37.897 | ............................................................*X | 37.897 | | 10.501 | ...............**X | 10.501 | | 4.500 | ......X | 4.500 | | 1.999 | ..X | 1.999 | | 0.999 | *X | 0.999 | | 0.000 | ***********..*..***X | 2.500 | | 1.500 | .X | 1.500 | | 0.500 | *.**..*.*X | 2.999 | | 2.499 | ...****X | 2.502 | | 2.501 | ...X | 2.501 | | 1.500 | .X | 1.500 | | 0.500 | *X | 0.500 | | 0.000 | ****************************************.*.*X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | *************.**.*..*X | 2.799 | | 2.699 | ...**.*..........*..................................**.................................**..................X | 61.980 | | 42.380 | ...................................................................*X | 42.380 | | 19.900 | ...............................X | 19.900 | | 2.200 | ..*X | 2.200 | | 0.000 | **********..**..*.......**.*..X | 9.003 | | 5.003 | .......**X | 5.003 | | 2.503 | ...X | 2.503 | | 0.000 | ****************.*..*.*X | 3.000 | | 2.699 | ...****X | 2.700 | | 2.301 | ..*X | 2.301 | | 2.300 | ..***.X | 3.000 | | 2.000 | ..X | 2.000 | | 1.000 | *X | 1.000 | | 0.000 | ******...*...**...*X | 6.005 | | 2.002 | ..*X | 2.002 | | 0.000 | ************.**..****X | 1.999 | | 1.000 | X | 1.000 | | 0.000 | **********************.*..*.*X | 2.999 | | 1.999 | ..*X | 1.999 | | 0.000 | ************************************..*..**....*X | 5.000 | | 4.001 | .....X | 4.001 | | 3.000 | ...*X | 3.000 | | 1.000 | X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ********************.**.X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ***...**..*.X | 4.000 | | 2.500 | ...*X | 2.500 | | 1.500 | .X | 1.500 | | 1.499 | .**..**X | 2.499 | | 1.501 | .*X | 1.501 | | 0.500 | X | 0.500 | | 0.000 | **********.**.*X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | **...*...*...*X | 5.998 | | 3.998 | .....X | 3.998 | | 2.501 | ...*X | 2.501 | | 1.001 | *.X | 1.501 | | 1.000 | *X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ***********************..**X | 1.501 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ************......*......**.......*X | 11.995 | | 7.995 | ...........X | 7.995 | | 4.000 | .....*X | 4.000 | | 1.500 | .**...*...X | 5.498 | | 4.498 | ......*X | 4.498 | | 2.498 | ...X | 2.498 | | 0.500 | *X | 0.500 | | 0.000 | **....***.................*....................*...........*X | 32.600 | | 28.098 | ............................................*X | 28.098 | | 13.992 | .....................**X | 13.992 | | 7.994 | ...........X | 7.994 | | 4.000 | .....X | 4.000 | | 0.000 | **....**........**X | 7.500 | | 5.000 | .......X | 5.000 | | 2.500 | ...X | 2.500 | | 0.000 | ********************************************************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 06:36:41,625 | test | INFO | pool-2-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **..*....................*....................*................X | 30.400 | | 19.900 | .....................................*X | 19.900 | | 9.400 | .................X | 9.400 | | 0.000 | ********************************************************************..**....*....*X | 5.500 | | 5.000 | ........*X | 5.000 | | 4.500 | .......*.**..X | 6.000 | | 5.500 | .........X | 5.500 | | 3.000 | ....*X | 3.000 | | 1.000 | *X | 1.000 | | 0.000 | ************...**....*....X | 6.000 | | 4.000 | ......*X | 4.000 | | 2.000 | ..X | 2.000 | | 1.000 | **..**..X | 2.999 | | 1.999 | ..X | 1.999 | | 1.499 | .***..*X | 2.502 | | 2.501 | ...*X | 2.501 | | 2.000 | ..X | 2.000 | | 1.500 | .**X | 1.500 | | 1.000 | *X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | *************************************.**.X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | *************.*....*....****.........................**.................**....X | 28.698 | | 28.691 | ......................................................*X | 28.691 | | 28.291 | .....................................................*X | 28.291 | | 19.301 | ....................................X | 19.301 | | 11.301 | ....................X | 11.301 | | 2.001 | ..*X | 2.001 | | 1.000 | X | 1.000 | | 0.000 | ***********...........**............*...........X | 18.007 | | 12.007 | ......................*X | 12.007 | | 6.007 | ..........*X | 6.007 | | 0.000 | *****************..**.**.........**..........*......*X | 15.007 | | 12.504 | .......................X | 12.504 | | 9.003 | ................*X | 9.003 | | 6.499 | ...........*X | 6.499 | | 2.000 | ..*X | 2.000 | | 0.000 | **.*..***..X | 2.998 | | 2.498 | ...X | 2.498 | | 2.000 | ..*X | 2.000 | | 1.501 | .*X | 1.501 | | 1.500 | .*X | 1.500 | | 1.500 | .*X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ***********.*.*X | 1.500 | | 1.000 | *X | 1.000 | | 0.000 | ****.**..*..*X | 3.001 | | 2.000 | ..X | 2.000 | | 1.000 | *X | 1.000 | | 0.000 | ***********.*.*X | 1.500 | | 0.999 | *X | 0.999 | | 0.000 | ********************************.**.**.*.**.*X | 3.000 | | 2.000 | ..*X | 2.000 | | 1.500 | .*X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ***********.*.*X | 1.501 | | 1.000 | X | 1.000 | | 0.500 | X | 0.500 | | 0.500 | **.****X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ***..**..X | 2.501 | | 2.500 | ...*.**....*.....**X | 7.497 | | 4.497 | .......X | 4.497 | | 2.002 | ..*X | 2.002 | | 1.000 | **.X | 1.500 | | 1.000 | *X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ****.*.*X | 1.500 | | 1.000 | *X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ***..*...**.....*X | 5.499 | | 4.999 | ........X | 4.999 | | 4.501 | .......*X | 4.501 | | 2.001 | ..X | 2.001 | | 1.001 | X | 1.001 | | 0.000 | ***********************.**....*..*..*X | 5.002 | | 3.000 | ....*X | 3.000 | | 1.500 | .X | 1.500 | | 0.000 | ****...*...*.*X | 4.000 | | 2.499 | ...X | 2.499 | | 1.000 | *X | 1.000 | | 0.500 | **....***X | 2.499 | | 1.499 | .X | 1.499 | | 0.500 | *X | 0.500 | | 0.000 | **.***.X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | *****.......................**.............................................................................**X | 52.000 | | 48.996 | .............................................................................................X | 48.996 | | 29.995 | ........................................................X | 29.995 | | 11.995 | ......................*X | 11.995 | | 3.999 | ......*****.X | 4.503 | | 3.000 | ....X | 3.000 | | 2.500 | ...*X | 2.500 | | 1.000 | X | 1.000 | | 0.500 | X | 0.500 | | 0.000 | ****.***.X | 1.500 | | 1.000 | X | 1.000 | | 0.500 | *X | 0.500 | | 0.000 | ******.*.*X | 1.500 | | 1.000 | **.*.*..*X | 3.000 | | 2.000 | ..X | 2.000 | | 1.000 | X | 1.000 | | 0.000 | *****************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-05-03 06:36:42,634 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................*.................................*.....................................*..........*X | 1059.700 | | 976.400 | ...........................................................................................X | 976.400 | | 812.500 | ...........................................................................*X | 812.500 | | 697.000 | ................................................................X | 697.000 | | 602.100 | .......................................................X | 602.100 | | 550.200 | ..................................................*X | 550.200 | | 526.500 | ................................................X | 526.500 | | 516.800 | ...............................................**X | 517.400 | | 495.600 | .............................................*X | 495.600 | | 470.200 | ...........................................*.*..**........*...**..X | 644.057 | | 589.903 | ......................................................X | 589.903 | | 564.401 | ....................................................*X | 564.401 | | 501.749 | ..............................................X | 501.749 | | 434.335 | .......................................X | 434.335 | | 363.735 | .................................X | 363.735 | | 332.187 | ..............................**..*.*X | 360.954 | | 360.445 | .................................X | 360.445 | | 322.845 | .............................X | 322.845 | | 309.945 | ............................****..*..*..**.....*....**.X | 487.449 | | 465.074 | ..........................................X | 465.074 | | 427.603 | .......................................**.***..*X | 459.166 | | 442.366 | ........................................*X | 442.366 | | 437.966 | ........................................X | 437.966 | | 420.280 | ......................................X | 420.280 | | 414.480 | ......................................**.*.**.....X | 490.729 | | 467.529 | ...........................................**X | 473.100 | | 407.100 | .....................................*X | 407.100 | | 388.600 | ...................................X | 388.600 | | 325.000 | .............................X | 325.000 | | 299.300 | ...........................*X | 299.300 | | 252.985 | ......................X | 252.985 | | 245.085 | ......................*X | 245.085 | | 236.285 | .....................**...*...*....**..****X | 367.187 | | 358.270 | ................................*.****..*X | 390.019 | | 388.619 | ...................................**..**...**..*...*X | 497.699 | | 485.399 | ............................................X | 485.399 | | 434.618 | ........................................X | 434.618 | | 369.897 | .................................*X | 369.897 | | 353.197 | ................................**..*...*X | 403.606 | | 394.233 | ....................................X | 394.233 | | 384.955 | ...................................***.*X | 396.978 | | 366.478 | .................................X | 366.478 | | 331.658 | ..............................X | 331.658 | | 291.934 | ..........................**.X | 305.934 | | 302.775 | ...........................**.*X | 314.771 | | 309.871 | ............................***...**.*....X | 399.300 | | 366.100 | .................................**..X | 387.900 | | 369.700 | .................................**.....*X | 413.900 | | 406.200 | .....................................*..X | 427.200 | | 412.500 | .....................................**.**.*.**..*.**.*X | 479.632 | | 466.800 | ...........................................**X | 466.800 | | 435.800 | ........................................X | 435.800 | | 401.600 | ....................................X | 401.600 | | 374.400 | ..................................*X | 374.400 | | 372.923 | ..................................**.X | 390.323 | | 382.923 | ...................................**X | 385.139 | | 352.339 | ................................*X | 352.339 | | 336.139 | ..............................X | 336.139 | | 303.429 | ...........................*X | 303.429 | | 280.229 | .........................X | 280.229 | | 279.729 | .........................X | 279.729 | | 268.900 | ........................**...X | 301.300 | | 299.621 | ...........................**..*X | 319.221 | | 271.666 | ........................*X | 271.666 | | 224.866 | ....................*X | 224.866 | | 211.203 | ..................X | 211.203 | | 200.937 | .................*....**....**.....*..**..*.*..****...*..**...X | 492.246 | | 483.942 | ............................................X | 483.942 | | 450.342 | .........................................*X | 450.342 | | 410.242 | .....................................*X | 410.242 | | 409.457 | .....................................X | 409.457 | | 375.957 | ..................................*X | 375.957 | | 359.557 | ................................X | 359.557 | | 323.714 | .............................**....**...**X | 392.771 | | 377.159 | ..................................*.X | 385.736 | | 385.431 | ...................................*X | 385.431 | | 381.155 | ..................................*X | 381.155 | | 366.212 | .................................**..**......*...***..X | 506.540 | | 449.320 | .........................................X | 449.320 | | 426.320 | .......................................*X | 426.320 | | 385.389 | ...................................*X | 385.389 | | 382.354 | ...................................*X | 382.354 | | 357.438 | ................................*****..*X | 372.362 | | 340.662 | ...............................X | 340.662 | | 320.054 | .............................X | 320.054 | | 313.955 | ............................**..***..**..*X | 381.478 | | 367.563 | .................................*X | 367.563 | | 337.563 | ..............................X | 337.563 | | 323.933 | .............................*.**..*..**.*...*..**..**.....**..*X | 531.438 | | 482.376 | ............................................X | 482.376 | | 414.576 | ......................................*X | 414.576 | | 348.938 | ...............................*X | 348.938 | | 342.859 | ...............................X | 342.859 | | 336.259 | ..............................*.**X | 342.400 | | 340.700 | ...............................***X | 343.200 | | 336.300 | ..............................X | 336.300 | | 325.700 | .............................*X | 325.700 | | 318.647 | .............................**....**..**.X | 397.387 | | 384.887 | ...................................**..**..*..X | 446.511 | | 444.038 | ........................................*X | 444.038 | | 436.227 | ........................................X | 436.227 | | 431.127 | .......................................*X | 431.127 | | 411.111 | .....................................*X | 411.111 | | 387.911 | ...................................X | 387.911 | | 352.911 | ................................*X | 352.911 | | 335.932 | ..............................*X | 335.932 | | 290.632 | ..........................**X | 290.632 | | 284.400 | .........................*....*.....***X | 376.600 | | 358.900 | ................................X | 358.900 | | 323.200 | .............................*..*X | 340.300 | | 330.048 | ..............................*.*X | 339.148 | | 321.034 | .............................**.*..**....*.*.**X | 422.982 | | 414.682 | ......................................*.X | 424.502 | | 402.314 | ....................................****.**X | 405.258 | | 397.058 | ....................................X | 397.058 | | 394.758 | ....................................X | 394.758 | | 378.737 | ..................................**.*X | 382.255 | | 376.644 | ..................................***X | 377.409 | | 376.709 | ..................................*.*X | 388.464 | | 374.181 | ..................................*X | 381.481 | | 366.500 | .................................**..*X | 382.300 | | 375.400 | ..................................*X | 379.151 | | 375.713 | ..................................**.*..**..**.X | 444.519 | | 436.944 | ........................................*X | 436.944 | | 430.502 | .......................................X | 430.502 | | 408.002 | .....................................X | 408.002 | | 389.722 | ...................................*X | 389.722 | | 376.109 | ..................................*X | 376.109 | | 352.309 | ................................X | 352.309 | | 350.795 | ................................*X | 350.795 | | 320.906 | .............................*X | 320.906 | | 305.862 | ...........................***X | 305.962 | | 299.562 | ...........................X | 299.562 | | 297.500 | ...........................****.***.X | 326.448 | | 323.048 | .............................**X | 325.648 | | 305.400 | ...........................**..*.*.*X | 346.884 | | 343.084 | ...............................*.**..**....**.*X | 432.400 | | 405.500 | .....................................**X | 405.500 | | 357.700 | ................................X | 357.700 | | 314.300 | ............................X | 314.300 | | 306.600 | ...........................**.*.X | 320.000 | | 289.100 | ..........................*X | 289.100 | | 286.100 | .........................**.*..**.*..*.**..**X | 379.527 | | 355.027 | ................................X | 355.027 | | 335.100 | ..............................**..*..**....**.***..*.....*..**..*X | 541.491 | | 470.145 | ...........................................*X | 470.145 | | 403.745 | .....................................*X | 403.745 | | 316.345 | ............................X | 316.345 | | 312.250 | ............................*X | 312.250 | | 282.450 | .........................**...X | 315.350 | | 305.716 | ...........................*...*X | 335.016 | | 329.116 | ..............................***...*X | 361.191 | | 358.491 | ................................*.X | 364.300 | | 323.800 | .............................*X | 323.800 | | 319.800 | .............................X | 319.800 | | 288.500 | ..........................**.*.*...**..**.....*..*.*X | 449.722 | | 414.300 | ......................................X | 414.300 | | 388.300 | ...................................*X | 388.300 | | 375.000 | ..................................**.*X | 391.800 | | 379.500 | ..................................*X | 379.500 | | 330.900 | ..............................X | 330.900 | | 318.500 | .............................X | 318.500 | | 302.000 | ...........................**..X | 318.673 | | 298.473 | ...........................**.***...X | 341.051 | | 332.751 | ..............................***.*..X | 364.165 | | 356.265 | ................................*X | 356.265 | | 341.165 | ...............................X | 341.165 | | 316.121 | ............................**...******....*...**..**..***.....*....*.*X | 562.062 | | 539.662 | .................................................*X | 539.662 | | 470.762 | ...........................................X | 470.762 | | 423.181 | ......................................*X | 423.181 | | 377.681 | ..................................X | 377.681 | | 372.735 | ..................................X | 372.735 | | 358.860 | ................................*X | 358.860 | | 348.673 | ...............................*X | 348.673 | | 285.582 | .........................**X | 285.582 | | 250.829 | ......................X | 250.829 | | 220.424 | ...................X | 220.424 | +---------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 06:36:44,786 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ | 172.23.98.196 | kv | 37.396049896 | 3693387776 | 2440921088 | 0 / 3909087232 | 168600 / 337360 | 7.0.0-5085-enterprise | | 172.23.98.195 | kv | 37.6400104194 | 3693387776 | 2424426496 | 0 / 3909087232 | 168802 / 336464 | 7.0.0-5085-enterprise | | 172.23.121.10 | kv | 49.2910543955 | 4201836544 | 2979450880 | 0 / 3758092288 | 168400 / 337600 | 7.0.0-5085-enterprise | +---------------+----------+-----------------+------------+------------+----------------+-------------------+-----------------------+ 2021-05-03 06:36:59,607 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 06:38:29,029 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 153190936 | 267232524 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 153135432 | 266197828 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 430562264 | 470670939 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-03 06:38:29,030 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-03 06:38:36,826 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load during rebalance_in 2021-05-03 06:38:36,828 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : rebalance_in 2021-05-03 06:38:36,953 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 06:38:44,338 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-05-03 06:38:56,078 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 21.5477996965 | Cluster node | | 172.23.98.195 | kv | 7.0.0-5085-enterprise | 12.8517110266 | Cluster node | | 172.23.121.10 | kv | 7.0.0-5085-enterprise | 9.45602440264 | Cluster node | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:38:56,148 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:39:02,096 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:39:07,487 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 1.96 2021-05-03 06:39:12,750 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 6.27 2021-05-03 06:39:17,974 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 10.36 2021-05-03 06:39:23,167 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 14.57 2021-05-03 06:39:28,367 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 18.83 2021-05-03 06:39:33,588 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 22.5 2021-05-03 06:39:38,710 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 25.64 2021-05-03 06:39:43,908 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 29.55 2021-05-03 06:39:49,072 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-03 06:39:54,250 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 34.16 2021-05-03 06:39:59,499 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 38.65 2021-05-03 06:40:04,648 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 42.05 2021-05-03 06:40:09,822 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 45.03 2021-05-03 06:40:14,969 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 48.98 2021-05-03 06:40:20,240 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 52.89 2021-05-03 06:40:25,530 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 56.2 2021-05-03 06:40:30,730 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 59.77 2021-05-03 06:40:36,000 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 63.1 2021-05-03 06:40:43,377 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-03 06:40:44,930 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:scope1:i-56-572000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-356 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:45,010 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:_default:2ycb5HZE1yRG5vNgM6-56-551000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-356 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:45,056 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:_default:AU0_7f0vhq-56-552000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-424 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:45,163 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:pHRlTFttn3E-4-56-561000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-424 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:46,766 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:Ch%z7dUUqg-LvWf-56-560000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-104 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:47,345 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:_default:Ch%z7dUUqg-LvWf-56-560000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:48,029 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:i-56-572000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:48,039 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:2ycb5HZE1yRG5vNgM6-56-551000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:48,756 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 71.98 2021-05-03 06:40:50,894 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:scope1:i-56-572000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-405 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-479 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:51,039 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:_default:2ycb5HZE1yRG5vNgM6-56-551000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-405 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-479 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:51,184 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:pHRlTFttn3E-4-56-561000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2794 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:51,187 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:pHRlTFttn3E-4-56-561000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2794 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:51,390 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:rs--43-291000:1rVQ3PWsM4wV-ss4Ezme-43-481000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:51,391 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:rs--43-291000:1rVQ3PWsM4wV-ss4Ezme-43-481000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1542 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:53,325 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2681 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2595 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:53,328 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2649 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2681 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2595 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:53,331 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:_default:2ycb5HZE1yRG5vNgM6-56-551000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2571 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:53,332 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:_default:2ycb5HZE1yRG5vNgM6-56-551000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2649 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2571 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:53,944 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 75.11 2021-05-03 06:40:54,486 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-616 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-636 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-502 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-588 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-595 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-562 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-585 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:54,743 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:Ch%z7dUUqg-LvWf-56-560000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2776 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2756 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2872 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:54,744 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:Ch%z7dUUqg-LvWf-56-560000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2776 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2756 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2872 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:54,746 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-58 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-144 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-166 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-123 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:54,766 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:Ch%z7dUUqg-LvWf-56-560000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-206 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-377 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-367 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-210 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-332 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-396 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-250 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-361 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-380 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:56,536 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:56,539 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-957 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:56,542 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:_default:Ch%z7dUUqg-LvWf-56-560000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-957 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-912 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:56,546 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1701 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1691 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:56,546 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1701 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1691 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:56,884 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1555 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1568 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:56,884 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1555 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1568 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:59,198 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 78.65 2021-05-03 06:40:59,380 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-558 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-654 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-597 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-642 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-587 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-585 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:59,828 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:FHkokJltpxnx-56-574000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-175 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:59,888 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:_default:FhF7HphJUsFuN-56-550000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-175 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-163 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:59,901 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:FhF7HphJUsFuN-56-550000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2577 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:40:59,903 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:FhF7HphJUsFuN-56-550000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2577 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:00,224 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:_default:OjKOu2IX-56-563000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-636 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-595 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-562 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-585 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:00,374 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:scope1:FHkokJltpxnx-56-574000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-817 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-845 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-758 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-800 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-897 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:01,240 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2828 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:01,240 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:_default:TO%xNNWVp6U1PRUTTRO-56-551000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2828 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:01,242 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2828 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2767 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:01,243 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:_default:TO%xNNWVp6U1PRUTTRO-56-551000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2828 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2767 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:01,980 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-759 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-845 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-758 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-800 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:02,121 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-246 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-243 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-242 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:02,125 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:Ch%z7dUUqg-LvWf-56-560000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-410 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:03,818 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1630 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:03,819 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1753 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1630 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:03,933 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1117 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1087 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1142 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:03,934 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1049 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1117 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1142 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1021 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1087 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:04,114 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1928 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:04,115 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1928 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:04,430 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 82.24 2021-05-03 06:41:06,936 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:FHkokJltpxnx-56-574000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-339 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-328 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-316 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-334 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-286 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:06,937 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:_default:FhF7HphJUsFuN-56-550000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-339 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-328 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-316 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-334 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-286 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-264 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:07,002 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-647 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-601 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-644 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-598 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:07,046 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:FhF7HphJUsFuN-56-550000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:07,049 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:FhF7HphJUsFuN-56-550000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2888 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2869 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:08,494 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:SFQ1GloUKHFov-56-607000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-647 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-601 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-525 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-661 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:08,628 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:scope1:FHkokJltpxnx-56-574000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-909 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:09,372 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:gxNNs83S66O-56-552000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-622 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-501 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:09,618 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 85.97 2021-05-03 06:41:09,687 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-403 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-464 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-486 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-491 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:09,701 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:_default:a%mMVU_H_3YUxZ3-56-559000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-935 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-911 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-954 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:10,351 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:_default:2ycb5HZE1yRG5vNgM6-56-551000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2708 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2871 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:10,352 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:_default:2ycb5HZE1yRG5vNgM6-56-551000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2708 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2871 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2761 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:11,223 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1977 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:11,223 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1977 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:11,549 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:11,549 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2137 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2153 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:11,552 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1309 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1316 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1372 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1286 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1299 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:11,555 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1309 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1316 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1372 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1286 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1299 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:13,262 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-129 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-30 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:13,500 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2601 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:13,503 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2601 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:13,815 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:scope1:FHkokJltpxnx-56-574000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-461 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-495 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:13,825 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:_default:FhF7HphJUsFuN-56-550000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-474 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-461 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-495 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:14,756 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 89.27 2021-05-03 06:41:14,977 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:SFQ1GloUKHFov-56-607000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-709 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-779 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-744 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-766 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-890 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:15,219 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:gxNNs83S66O-56-552000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-879 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-744 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-887 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-766 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-890 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:16,154 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:rs--43-291000:1rVQ3PWsM4wV-ss4Ezme-43-481000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1720 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:16,155 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:rs--43-291000:1rVQ3PWsM4wV-ss4Ezme-43-481000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1720 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:16,961 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:oU0RIla5a7M-42-719000:Y3rIkI-42-890000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1758 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:16,963 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:oU0RIla5a7M-42-719000:Y3rIkI-42-890000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1758 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:17,051 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-766 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-863 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:17,272 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-17 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-84 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-52 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-102 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-184 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-141 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:17,345 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2579 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:17,346 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2579 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:17,757 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:_default:OjKOu2IX-56-563000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-804 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-863 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:18,171 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2315 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2285 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:18,171 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2315 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2285 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:18,657 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-739 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-704 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-744 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-766 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:19,220 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-238 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-355 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:19,877 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 91.81 2021-05-03 06:41:20,704 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:_default:f6AaSHeXFEeEmbZbZq%S-56-562000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-48 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-21 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-84 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-179 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:20,707 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-48 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-84 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-179 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:20,720 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:_default:f6AaSHeXFEeEmbZbZq%S-56-562000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2630 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:20,721 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:_default:f6AaSHeXFEeEmbZbZq%S-56-562000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2630 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:20,918 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-569 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-620 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-691 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:21,141 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-627 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-515 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-620 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-550 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-691 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:21,191 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-88 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-84 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-179 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:21,273 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:21,275 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2699 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:21,575 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:21,575 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2699 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:21,783 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:gxNNs83S66O-56-552000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-905 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-956 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:22,269 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in default:oU0RIla5a7M-42-719000:DV-42-931000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1723 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:22,272 | test | INFO | pool-2-thread-15 | [table_view:display:72] Keys failed in default:oU0RIla5a7M-42-719000:DV-42-931000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1723 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:22,584 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-329 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-355 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-310 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-271 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-280 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:23,262 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:OjKOu2IX-56-563000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-87 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-117 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-188 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:23,282 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2819 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2880 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2751 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:23,283 | test | INFO | pool-2-thread-25 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2819 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2738 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2880 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2751 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:23,533 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:oU0RIla5a7M-42-719000:Y3rIkI-42-890000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:23,535 | test | INFO | pool-2-thread-26 | [table_view:display:72] Keys failed in default:oU0RIla5a7M-42-719000:Y3rIkI-42-890000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1933 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:23,891 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1432 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:23,892 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1432 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:24,880 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-968 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:25,076 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 94.96 2021-05-03 06:41:25,298 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-451 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:25,299 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2706 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2747 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2702 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2752 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:25,301 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2706 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2747 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2702 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2873 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2752 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:25,405 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:_default:f6AaSHeXFEeEmbZbZq%S-56-562000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:25,417 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:26,757 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-747 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-876 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-861 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,052 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-747 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-876 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-861 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,144 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-338 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-314 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-233 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-284 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,210 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2743 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,211 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2743 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,315 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:DkhV4g2IJjYDgdV-56-550000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-696 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-541 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,345 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-434 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,470 | test | INFO | pool-2-thread-30 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:SFQ1GloUKHFov-56-607000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-968 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-951 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,611 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:_default:OjKOu2IX-56-563000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:27,611 | test | INFO | pool-2-thread-20 | [table_view:display:72] Keys failed in default:_default:OjKOu2IX-56-563000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2671 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:28,079 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:OjKOu2IX-56-563000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-219 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-258 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:28,157 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:_default:DkhV4g2IJjYDgdV-56-550000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:28,990 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2151 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:28,993 | test | INFO | pool-2-thread-1 | [table_view:display:72] Keys failed in default:vH9-43-755000:eyri-44-105000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2151 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:30,210 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:30,217 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 98.06 2021-05-03 06:41:30,328 | test | INFO | pool-2-thread-12 | [table_view:display:72] Keys failed in default:_default:f6AaSHeXFEeEmbZbZq%S-56-562000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-416 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-475 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-453 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:30,332 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-416 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-475 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-453 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:30,428 | test | INFO | pool-2-thread-4 | [table_view:display:72] Keys failed in default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-959 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-963 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:31,434 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-107 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-51 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:31,611 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-416 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-453 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,161 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:scope1:a-v9lrxgF7yl-56-569000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,204 | test | INFO | pool-2-thread-22 | [table_view:display:72] Keys failed in default:_default:DkhV4g2IJjYDgdV-56-550000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-806 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-836 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-733 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-873 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,221 | test | INFO | pool-2-thread-19 | [table_view:display:72] Keys failed in default:_default:G4NgvDNE_p1m-56-563000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-959 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,226 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2539 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,226 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2539 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,335 | test | INFO | pool-2-thread-13 | [table_view:display:72] Keys failed in default:_default:OjKOu2IX-56-563000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-416 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-453 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,632 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2413 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2583 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,634 | test | INFO | pool-2-thread-7 | [table_view:display:72] Keys failed in default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2413 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2583 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:32,936 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:scope1:a-v9lrxgF7yl-56-569000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-51 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-142 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:33,000 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:scope1:a-v9lrxgF7yl-56-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2583 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:33,003 | test | INFO | pool-2-thread-5 | [table_view:display:72] Keys failed in default:scope1:a-v9lrxgF7yl-56-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2583 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2640 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:33,671 | test | INFO | pool-2-thread-3 | [table_view:display:72] Keys failed in default:_default:DkhV4g2IJjYDgdV-56-550000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-358 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-311 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-364 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-321 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-281 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:33,740 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:33,742 | test | INFO | pool-2-thread-17 | [table_view:display:72] Keys failed in default:vH9-43-755000:PxCYkH30ichVOO-43-915000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1722 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:34,177 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:rs--43-291000:1rVQ3PWsM4wV-ss4Ezme-43-481000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2092 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2102 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:34,178 | test | INFO | pool-2-thread-29 | [table_view:display:72] Keys failed in default:rs--43-291000:1rVQ3PWsM4wV-ss4Ezme-43-481000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2092 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2164 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2102 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:34,694 | test | INFO | pool-2-thread-8 | [table_view:display:72] Keys failed in default:_default:UIVhNKt-56-551000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-29 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-165 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-181 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:34,734 | test | INFO | pool-2-thread-28 | [table_view:display:72] Keys failed in default:_default:UIVhNKt-56-551000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-629 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-617 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-577 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-687 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-532 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:34,773 | test | INFO | pool-2-thread-24 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-717 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-787 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-831 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:35,148 | test | INFO | pool-2-thread-21 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-249 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-228 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-305 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-340 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-295 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:35,398 | test | INFO | pool-2-thread-11 | [table_view:display:72] Keys failed in default:_default:3U1Y-56-551000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-687 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:35,401 | test | INFO | pool-2-thread-6 | [table_view:display:72] Keys failed in default:_default:29IrDQ-U7l1V-56-555000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-687 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:35,596 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-03 06:41:35,617 | test | INFO | pool-2-thread-9 | [table_view:display:72] Keys failed in default:scope1:a-v9lrxgF7yl-56-569000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-717 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-831 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:36,016 | test | INFO | pool-2-thread-10 | [table_view:display:72] Keys failed in default:scope1:a-v9lrxgF7yl-56-569000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-249 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-305 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-340 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-295 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:37,897 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:37,898 | test | INFO | pool-2-thread-18 | [table_view:display:72] Keys failed in default:_default:0-56-559000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2728 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-03 06:41:45,766 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:41:45,825 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 169.749000072 sec 2021-05-03 06:43:19,707 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 8 docs from 0 to 5000 2021-05-03 06:44:40,874 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 14 docs from 0 to 5000 2021-05-03 06:44:40,875 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 5 docs from 0 to 5000 2021-05-03 06:44:40,877 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 9 docs from 0 to 5000 2021-05-03 06:44:40,877 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 06:44:40,878 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 2 docs from 0 to 5000 2021-05-03 06:44:40,884 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 06:44:40,885 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 06:44:40,888 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 0 to 500 2021-05-03 06:44:40,890 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 06:44:40,891 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 06:44:40,891 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 06:44:40,900 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 15 docs from 500 to 980 2021-05-03 06:44:40,901 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 4 docs from 0 to 500 2021-05-03 06:44:40,901 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-03 06:44:40,903 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 06:44:40,904 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 16 docs from 0 to 500 2021-05-03 06:44:40,904 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 06:44:40,904 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 29 docs from 500 to 980 2021-05-03 06:44:40,905 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 23 docs from 0 to 500 2021-05-03 06:44:40,905 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 5 docs from 2500 to 2900 2021-05-03 06:44:40,907 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 5 docs from 2500 to 2900 2021-05-03 06:44:40,907 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 15 docs from 500 to 980 2021-05-03 06:44:40,907 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 20 docs from 0 to 500 2021-05-03 06:44:40,908 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-03 06:44:40,908 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 19 docs from 0 to 500 2021-05-03 06:44:40,921 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 3 docs from 2500 to 2900 2021-05-03 06:44:40,928 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 06:44:40,928 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 17 docs from 500 to 980 2021-05-03 06:44:40,930 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 23 docs from 500 to 980 2021-05-03 06:44:40,931 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 22 docs from 500 to 980 2021-05-03 06:44:40,957 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 19 docs from 500 to 980 2021-05-03 06:44:40,957 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-03 06:44:40,960 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 5 docs from 2500 to 2900 2021-05-03 06:44:40,960 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 06:44:40,961 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 18 docs from 0 to 500 2021-05-03 06:44:40,963 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 06:44:40,963 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 18 docs from 0 to 500 2021-05-03 06:44:40,987 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 6 docs from 2500 to 2900 2021-05-03 06:44:40,987 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 12 docs from 500 to 980 2021-05-03 06:44:40,989 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 11 docs from 0 to 500 2021-05-03 06:44:40,990 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 06:44:40,990 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 15 docs from 0 to 500 2021-05-03 06:44:40,990 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 06:44:40,990 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 2500 to 2900 2021-05-03 06:44:40,992 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 8 docs from 0 to 500 2021-05-03 06:44:40,992 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 06:44:40,993 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 11 docs from 500 to 980 2021-05-03 06:44:40,993 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 7 docs from 500 to 980 2021-05-03 06:44:40,994 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 12 docs from 0 to 500 2021-05-03 06:44:40,994 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 06:44:40,996 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 6 docs from 0 to 500 2021-05-03 06:44:40,996 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 2 docs from 2500 to 2900 2021-05-03 06:44:40,996 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 6 docs from 500 to 980 2021-05-03 06:44:40,997 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 6 docs from 0 to 500 2021-05-03 06:44:40,999 | test | ERROR | pool-2-thread-27 | [task:call:3825] Failed to load 1 docs from 500 to 980 2021-05-03 06:50:43,219 | test | INFO | pool-2-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****************************************************************************.**...........*...............................**................**....................**X | 181.000 | | 173.000 | ..........................................................................X | 173.000 | | 168.781 | .........................................................................*..........**....**........**....*X | 227.792 | | 218.963 | ...............................................................................................X | 218.963 | | 197.633 | .....................................................................................*X | 197.633 | | 180.833 | ..............................................................................X | 180.833 | | 165.021 | .......................................................................X | 165.021 | | 117.188 | ..................................................*X | 117.188 | | 93.988 | ........................................*X | 93.988 | | 69.800 | .............................X | 69.800 | | 8.400 | ..*X | 8.400 | | 0.000 | **********X | 1.799 | | 1.399 | X | 1.399 | | 1.199 | *X | 1.199 | | 0.600 | X | 0.600 | | 0.200 | *X | 0.200 | | 0.000 | ***************************...**.........*..............................**.......*X | 113.679 | | 113.062 | ................................................**X | 113.062 | | 110.188 | ...............................................X | 110.188 | | 107.519 | ..............................................**X | 107.719 | | 106.528 | .............................................*X | 106.528 | | 105.814 | .............................................X | 105.814 | | 101.714 | ...........................................*...*X | 107.900 | | 107.800 | ..............................................X | 107.800 | | 99.800 | ..........................................*X | 99.800 | | 87.000 | .....................................*X | 87.000 | | 78.600 | .................................**X | 78.600 | | 78.000 | .................................X | 78.000 | | 74.100 | ...............................X | 74.100 | | 52.900 | ......................**.**.........*X | 77.100 | | 74.700 | ...............................**..X | 77.700 | | 64.400 | ...........................X | 64.400 | | 63.000 | ..........................*X | 63.000 | | 58.400 | ........................**......*..**...*X | 83.184 | | 80.213 | ..................................*X | 80.213 | | 69.306 | .............................X | 69.306 | | 46.206 | ...................*X | 46.206 | | 41.193 | .................*X | 41.193 | | 38.804 | ................X | 38.804 | | 17.204 | ......*X | 17.204 | | 10.404 | ...X | 10.404 | | 6.600 | .X | 6.600 | | 3.800 | **X | 3.800 | | 3.300 | X | 3.300 | | 2.999 | X | 2.999 | | 1.499 | *X | 1.499 | | 0.999 | *X | 0.999 | | 0.400 | **...**.......*.................***X | 63.690 | | 61.890 | ..........................*...*...**.X | 77.912 | | 65.000 | ...........................*X | 65.000 | | 61.600 | ..........................**.*..X | 69.000 | | 67.900 | ............................*X | 67.900 | | 63.500 | ..........................*.**..**...*...........*...**...X | 115.859 | | 115.269 | .................................................*X | 115.269 | | 109.869 | ...............................................*X | 109.869 | | 82.526 | ...................................*X | 82.526 | | 75.826 | ................................**.***..X | 83.292 | | 82.992 | ...................................X | 82.992 | | 76.406 | ................................*X | 76.406 | | 69.606 | .............................*X | 69.606 | | 62.906 | ..........................*.....*...**...*...*X | 94.193 | | 94.093 | ........................................*X | 94.093 | | 73.877 | ...............................**.**...*......X | 96.737 | | 90.437 | ......................................*X | 90.437 | | 82.937 | ...................................X | 82.937 | | 75.500 | ................................**....**...*.X | 95.294 | | 88.294 | .....................................*X | 88.294 | | 84.094 | ...................................*.****...X | 91.408 | +---------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 06:50:44,778 | test | INFO | pool-2-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****X | 0.800 | | 0.400 | X | 0.400 | | 0.000 | **************************************************************************.**.*X | 4.600 | | 3.100 | X | 3.100 | | 1.900 | *X | 1.900 | | 1.800 | *X | 1.900 | | 1.200 | *X | 1.200 | | 0.700 | *X | 0.700 | | 0.300 | X | 0.300 | | 0.000 | **********X | 1.900 | | 1.400 | X | 1.400 | | 0.800 | *X | 0.800 | | 0.400 | *X | 0.400 | | 0.300 | ****X | 1.200 | | 0.900 | *X | 0.900 | | 0.700 | *...*..................**...............**..................*..........................................**....X | 208.728 | | 203.081 | ................................................................................................X | 203.081 | | 200.821 | ...............................................................................................****..X | 204.992 | | 186.172 | ........................................................................................*.*X | 187.972 | | 186.627 | ........................................................................................**.......X | 202.027 | | 191.827 | ..........................................................................................*X | 191.827 | | 183.385 | ......................................................................................X | 183.385 | | 152.185 | .......................................................................X | 152.185 | | 136.380 | ................................................................*X | 136.380 | | 105.895 | .................................................X | 105.895 | | 16.395 | ......*X | 16.395 | | 0.900 | **.***.X | 5.001 | | 3.901 | X | 3.901 | | 2.101 | *X | 2.101 | | 1.200 | *X | 1.200 | | 1.000 | ***X | 1.100 | | 0.900 | *X | 0.900 | | 0.300 | X | 0.300 | | 0.100 | ******..*........**....**..........***..X | 56.300 | | 52.000 | .......................*.*X | 53.600 | | 53.200 | ........................**..**X | 57.300 | | 53.300 | ........................X | 53.300 | | 48.500 | ......................X | 48.500 | | 43.300 | ...................**..**.......***X | 61.388 | | 60.988 | ............................X | 60.988 | | 51.788 | .......................**X | 51.788 | | 51.512 | .......................*.*.....*X | 63.700 | | 60.688 | ............................**......***....X | 82.312 | | 79.100 | ....................................X | 79.100 | | 41.500 | ..................*X | 41.500 | | 25.700 | ...........X | 25.700 | | 16.698 | ......X | 16.698 | | 2.998 | **......**.............*......................................**.....X | 131.764 | | 123.418 | ..........................................................X | 123.418 | | 85.288 | .......................................**X | 85.288 | | 77.088 | ...................................X | 77.088 | | 74.788 | ..................................X | 74.788 | | 74.612 | ..................................*.**..**........*..*X | 101.100 | | 95.900 | ............................................*......**...**.......*X | 128.000 | | 118.400 | .......................................................**.*.....**...........X | 154.187 | | 153.251 | ........................................................................X | 153.251 | | 132.366 | ..............................................................**X | 132.366 | | 112.566 | ....................................................X | 112.566 | | 104.828 | .................................................X | 104.828 | | 98.032 | .............................................*X | 98.032 | | 97.432 | .............................................**.........****.X | 117.989 | | 95.312 | ............................................**..*X | 98.612 | | 91.812 | ..........................................*.X | 92.500 | | 80.500 | .....................................*X | 80.500 | | 73.696 | ..................................X | 73.696 | | 50.296 | .......................****.....**X | 61.484 | | 56.884 | ..........................X | 56.884 | | 56.479 | ..........................***..X | 61.722 | | 61.622 | ............................X | 61.622 | | 57.200 | ..........................*X | 57.200 | | 56.200 | .........................*X | 56.200 | | 53.500 | ........................*..*X | 58.195 | | 57.495 | ..........................*X | 57.495 | | 46.303 | .....................*X | 46.303 | | 38.503 | .................*X | 38.503 | | 28.303 | ............X | 28.303 | | 3.900 | X | 3.900 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-03 06:50:45,667 | test | INFO | pool-2-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +----------+---------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........*............*...X | 395.071 | | 392.671 | .......................*X | 392.671 | | 208.000 | ...........X | 208.000 | | 85.800 | ....X | 85.800 | | 0.000 | ***..........*........*..............*X | 529.574 | | 442.452 | ..........................X | 442.452 | | 431.326 | .........................X | 431.326 | | 409.926 | ........................*...........*.......*X | 693.367 | | 681.731 | .........................................X | 681.731 | | 583.846 | ...................................X | 583.846 | | 573.006 | ..................................*...**...*X | 664.242 | | 591.570 | ...................................X | 591.570 | | 548.253 | .................................*....**..X | 649.653 | | 632.513 | ......................................X | 632.513 | | 551.113 | .................................X | 551.113 | | 492.513 | .............................X | 492.513 | | 489.385 | .............................X | 489.385 | | 467.685 | ............................X | 467.685 | | 457.715 | ...........................*X | 457.715 | | 387.906 | .......................*X | 387.906 | | 322.106 | ...................X | 322.106 | | 319.195 | ..................*.*.....X | 409.784 | | 386.238 | .......................*..X | 430.286 | | 428.364 | .........................**..*..X | 483.486 | | 438.586 | ..........................*.*.....**....*.*........**.*....*................**......*X | 1173.639 | | 1170.941 | .......................................................................**X | 1170.941 | | 1109.422 | ....................................................................X | 1109.422 | | 1047.222 | ................................................................X | 1047.222 | | 985.322 | ............................................................X | 985.322 | | 985.322 | ............................................................X | 985.322 | | 933.200 | .........................................................X | 933.200 | | 889.778 | ......................................................X | 889.778 | | 771.386 | ..............................................*X | 771.386 | | 766.186 | ..............................................*X | 766.186 | | 745.408 | .............................................X | 745.408 | | 627.370 | ......................................*X | 627.370 | | 590.170 | ...................................X | 590.170 | | 573.570 | ..................................X | 573.570 | | 543.900 | ................................*X | 543.900 | | 543.300 | ................................**.*.....*..**.X | 680.688 | | 648.685 | .......................................*X | 648.685 | | 602.785 | ....................................*X | 602.785 | | 551.997 | .................................X | 551.997 | | 483.193 | .............................**X | 483.293 | | 464.955 | ...........................X | 464.955 | | 437.961 | ..........................*..**.........*......................**....X | 1039.306 | | 1038.600 | ...............................................................**..........*....**......X | 1355.280 | | 1199.411 | .........................................................................X | 1199.411 | | 1170.911 | .......................................................................*X | 1170.911 | | 1139.713 | .....................................................................*........X | 1258.601 | | 1206.401 | ..........................................................................*X | 1206.401 | | 1104.887 | ...................................................................X | 1104.887 | | 740.424 | .............................................*X | 740.424 | | 675.224 | .........................................*X | 675.224 | | 607.337 | ....................................X | 607.337 | | 501.947 | ..............................**X | 511.847 | | 494.487 | .............................*.*X | 509.741 | | 460.441 | ...........................*X | 460.441 | | 404.863 | ........................**.**..*....**.....**........*.................................*.........**....*.........*X | 1607.222 | | 1605.266 | ..................................................................................................*X | 1605.266 | | 1500.866 | ............................................................................................X | 1500.866 | | 1009.466 | .............................................................X | 1009.466 | | 834.392 | ..................................................*X | 834.392 | | 733.303 | ............................................X | 733.303 | | 602.803 | ....................................**.*X | 614.838 | | 612.513 | .....................................X | 612.513 | | 611.613 | .....................................*X | 611.613 | | 556.441 | .................................X | 556.441 | | 554.254 | .................................*.....**....*X | 704.300 | | 552.000 | .................................*X | 552.000 | | 461.200 | ...........................*X | 461.200 | | 401.000 | .......................X | 401.000 | | 294.900 | .................*X | 294.900 | | 291.300 | .................****X | 294.500 | | 289.800 | .................X | 289.800 | | 288.300 | ................*..**..*X | 338.100 | | 270.700 | ...............*X | 270.700 | | 263.600 | ...............**X | 266.900 | | 236.000 | .............*X | 236.000 | | 207.064 | ...........**...**.**..*..****.***.*....*...........**......**.....**X | 775.009 | | 720.309 | ...........................................X | 720.309 | | 657.853 | .......................................X | 657.853 | | 465.736 | ...........................*X | 465.736 | | 421.436 | .........................*X | 421.436 | | 384.842 | ......................*X | 384.842 | | 370.142 | ......................X | 370.142 | | 370.142 | ......................X | 370.142 | | 314.442 | ..................X | 314.442 | | 194.635 | ...........****...*.....X | 329.200 | | 318.300 | ..................*X | 318.300 | | 313.600 | ..................**..*.*X | 361.300 | | 336.600 | ...................X | 336.600 | | 266.600 | ...............*X | 266.600 | | 252.800 | ..............**.**.**.*.....*X | 377.002 | | 364.902 | .....................X | 364.902 | | 326.564 | ...................X | 326.564 | | 230.875 | .............****..*...X | 313.969 | | 313.869 | ..................***.***X | 327.631 | | 316.100 | ..................**..***.*X | 367.374 | | 322.374 | ...................X | 322.374 | | 320.725 | ..................X | 320.725 | | 300.004 | .................**.*X | 310.004 | | 285.979 | ................X | 285.979 | | 250.805 | ..............X | 250.805 | | 246.305 | ..............**.*..***X | 303.400 | | 270.282 | ...............*X | 270.282 | | 245.782 | ..............*X | 245.782 | | 217.782 | ............X | 217.782 | | 152.410 | ........*X | 152.410 | | 121.810 | ......*X | 121.810 | | 89.410 | ....X | 89.410 | | 19.300 | X | 19.300 | +----------+---------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-03 06:50:46,644 | test | WARNING | Thread-2691 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:TO%xNNWVp6U1PRUTTRO-56-551000 create failed keys from task: [u'test_collections-2828', u'test_collections-2767'] 2021-05-03 06:50:46,645 | test | WARNING | Thread-2699 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:FhF7HphJUsFuN-56-550000 delete failed keys from task: [u'test_collections-98', u'test_collections-328', u'test_collections-348', u'test_collections-126', u'test_collections-461', u'test_collections-286', u'test_collections-264', u'test_collections-163', u'test_collections-2', u'test_collections-339', u'test_collections-316', u'test_collections-159', u'test_collections-85', u'test_collections-116', u'test_collections-259', u'test_collections-334', u'test_collections-474', u'test_collections-153', u'test_collections-175', u'test_collections-495'] 2021-05-03 06:50:46,648 | test | WARNING | Thread-2703 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:AU0_7f0vhq-56-552000 delete failed keys from task: [u'test_collections-424'] 2021-05-03 06:50:46,648 | test | WARNING | Thread-2704 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:2ycb5HZE1yRG5vNgM6-56-551000 update failed keys from task: [u'test_collections-910'] 2021-05-03 06:50:46,650 | test | WARNING | Thread-2710 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:OjKOu2IX-56-563000 update failed keys from task: [u'test_collections-804', u'test_collections-527', u'test_collections-522', u'test_collections-863', u'test_collections-567', u'test_collections-864', u'test_collections-562', u'test_collections-585', u'test_collections-581', u'test_collections-636', u'test_collections-614', u'test_collections-595', u'test_collections-651', u'test_collections-574', u'test_collections-772'] 2021-05-03 06:50:46,647 | test | WARNING | Thread-2701 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:G4NgvDNE_p1m-56-563000 update failed keys from task: [u'test_collections-627', u'test_collections-824', u'test_collections-747', u'test_collections-788', u'test_collections-861', u'test_collections-620', u'test_collections-959', u'test_collections-515', u'test_collections-876', u'test_collections-550', u'test_collections-691'] 2021-05-03 06:50:46,654 | test | WARNING | Thread-2712 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default update failed keys from task: [u'test_collections-968', u'test_collections-647', u'test_collections-601', u'test_collections-766', u'test_collections-602', u'test_collections-885', u'test_collections-644', u'test_collections-739', u'test_collections-931', u'test_collections-974', u'test_collections-697', u'test_collections-704', u'test_collections-744', u'test_collections-864', u'test_collections-519', u'test_collections-914', u'test_collections-757', u'test_collections-834', u'test_collections-516', u'test_collections-712', u'test_collections-798', u'test_collections-598', u'test_collections-871'] 2021-05-03 06:50:46,707 | test | WARNING | Thread-2704 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:2ycb5HZE1yRG5vNgM6-56-551000 delete failed keys from task: [u'test_collections-438', u'test_collections-405', u'test_collections-479', u'test_collections-356'] 2021-05-03 06:50:46,648 | test | WARNING | Thread-2705 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:3U1Y-56-551000 update failed keys from task: [u'test_collections-687'] 2021-05-03 06:50:46,650 | test | WARNING | Thread-2711 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:a%mMVU_H_3YUxZ3-56-559000 update failed keys from task: [u'test_collections-729', u'test_collections-845', u'test_collections-527', u'test_collections-800', u'test_collections-522', u'test_collections-962', u'test_collections-567', u'test_collections-562', u'test_collections-938', u'test_collections-616', u'test_collections-935', u'test_collections-614', u'test_collections-651', u'test_collections-574', u'test_collections-906', u'test_collections-868', u'test_collections-502', u'test_collections-943', u'test_collections-588', u'test_collections-585', u'test_collections-781', u'test_collections-581', u'test_collections-718', u'test_collections-759', u'test_collections-636', u'test_collections-911', u'test_collections-758', u'test_collections-954', u'test_collections-595'] 2021-05-03 06:50:46,788 | test | WARNING | Thread-2704 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:2ycb5HZE1yRG5vNgM6-56-551000 create failed keys from task: [u'test_collections-2708', u'test_collections-2649', u'test_collections-2571', u'test_collections-2871', u'test_collections-2761'] 2021-05-03 06:50:46,888 | test | WARNING | Thread-2701 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:G4NgvDNE_p1m-56-563000 delete failed keys from task: [u'test_collections-34', u'test_collections-346', u'test_collections-303', u'test_collections-284', u'test_collections-25', u'test_collections-88', u'test_collections-416', u'test_collections-84', u'test_collections-338', u'test_collections-314', u'test_collections-179', u'test_collections-155', u'test_collections-71', u'test_collections-233', u'test_collections-453', u'test_collections-152', u'test_collections-293', u'test_collections-8'] 2021-05-03 06:50:46,890 | test | WARNING | Thread-2710 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:OjKOu2IX-56-563000 delete failed keys from task: [u'test_collections-96', u'test_collections-346', u'test_collections-389', u'test_collections-188', u'test_collections-484', u'test_collections-25', u'test_collections-87', u'test_collections-416', u'test_collections-219', u'test_collections-83', u'test_collections-117', u'test_collections-258', u'test_collections-453', u'test_collections-152', u'test_collections-8'] 2021-05-03 06:50:46,999 | test | WARNING | Thread-2712 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default delete failed keys from task: [u'test_collections-10', u'test_collections-33', u'test_collections-129', u'test_collections-30', u'test_collections-300', u'test_collections-238', u'test_collections-355', u'test_collections-451', u'test_collections-471', u'test_collections-150', u'test_collections-290', u'test_collections-170'] 2021-05-03 06:50:47,411 | test | WARNING | Thread-2712 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default create failed keys from task: [u'test_collections-2819', u'test_collections-2601', u'test_collections-2738', u'test_collections-2880', u'test_collections-2751'] 2021-05-03 06:50:47,451 | test | WARNING | Thread-2699 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:FhF7HphJUsFuN-56-550000 create failed keys from task: [u'test_collections-2577', u'test_collections-2888', u'test_collections-2869'] 2021-05-03 06:50:47,461 | test | WARNING | Thread-2701 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:G4NgvDNE_p1m-56-563000 create failed keys from task: [u'test_collections-2699'] 2021-05-03 06:50:47,467 | test | WARNING | Thread-2807 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:i-56-572000 update failed keys from task: [u'test_collections-910'] 2021-05-03 06:50:47,469 | test | WARNING | Thread-2808 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:FHkokJltpxnx-56-574000 update failed keys from task: [u'test_collections-909', u'test_collections-906', u'test_collections-729', u'test_collections-845', u'test_collections-868', u'test_collections-800', u'test_collections-943', u'test_collections-781', u'test_collections-817', u'test_collections-718', u'test_collections-938', u'test_collections-758', u'test_collections-832', u'test_collections-797', u'test_collections-897'] 2021-05-03 06:50:47,470 | test | WARNING | Thread-2809 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:a-v9lrxgF7yl-56-569000 update failed keys from task: [u'test_collections-717', u'test_collections-867', u'test_collections-822', u'test_collections-831', u'test_collections-630', u'test_collections-586', u'test_collections-752'] 2021-05-03 06:50:47,476 | test | WARNING | Thread-2807 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:i-56-572000 delete failed keys from task: [u'test_collections-438', u'test_collections-405', u'test_collections-479', u'test_collections-356'] 2021-05-03 06:50:47,494 | test | WARNING | Thread-2809 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:a-v9lrxgF7yl-56-569000 delete failed keys from task: [u'test_collections-249', u'test_collections-51', u'test_collections-305', u'test_collections-142', u'test_collections-340', u'test_collections-295'] 2021-05-03 06:50:47,496 | test | WARNING | Thread-2710 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:OjKOu2IX-56-563000 create failed keys from task: [u'test_collections-2671'] 2021-05-03 06:50:47,506 | test | WARNING | Thread-2880 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:rs--43-291000:1rVQ3PWsM4wV-ss4Ezme-43-481000 create failed keys from task: [u'test_collections-2092', u'test_collections-1720', u'test_collections-2164', u'test_collections-1542', u'test_collections-2102'] 2021-05-03 06:50:47,509 | test | WARNING | Thread-2711 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:a%mMVU_H_3YUxZ3-56-559000 delete failed keys from task: [u'test_collections-58', u'test_collections-403', u'test_collections-203', u'test_collections-401', u'test_collections-246', u'test_collections-166', u'test_collections-243', u'test_collections-123', u'test_collections-486', u'test_collections-164', u'test_collections-362', u'test_collections-121', u'test_collections-242', u'test_collections-90', u'test_collections-491', u'test_collections-468', u'test_collections-144', u'test_collections-189', u'test_collections-464', u'test_collections-2', u'test_collections-474', u'test_collections-393', u'test_collections-7'] 2021-05-03 06:50:47,510 | test | WARNING | Thread-2809 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:a-v9lrxgF7yl-56-569000 create failed keys from task: [u'test_collections-2583', u'test_collections-2640'] 2021-05-03 06:50:47,516 | test | WARNING | Thread-2718 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:UIVhNKt-56-551000 update failed keys from task: [u'test_collections-629', u'test_collections-617', u'test_collections-577', u'test_collections-687', u'test_collections-641', u'test_collections-532'] 2021-05-03 06:50:47,519 | test | WARNING | Thread-2722 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DkhV4g2IJjYDgdV-56-550000 update failed keys from task: [u'test_collections-806', u'test_collections-726', u'test_collections-549', u'test_collections-843', u'test_collections-720', u'test_collections-541', u'test_collections-618', u'test_collections-836', u'test_collections-733', u'test_collections-696', u'test_collections-873'] 2021-05-03 06:50:47,558 | test | WARNING | Thread-2718 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:UIVhNKt-56-551000 delete failed keys from task: [u'test_collections-29', u'test_collections-12', u'test_collections-42', u'test_collections-133', u'test_collections-165', u'test_collections-181'] 2021-05-03 06:50:47,609 | test | WARNING | Thread-2769 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:79vO7vaBVKc-56-599000:SFQ1GloUKHFov-56-607000 update failed keys from task: [u'test_collections-968', u'test_collections-647', u'test_collections-601', u'test_collections-766', u'test_collections-525', u'test_collections-602', u'test_collections-885', u'test_collections-779', u'test_collections-697', u'test_collections-890', u'test_collections-709', u'test_collections-744', u'test_collections-864', u'test_collections-661', u'test_collections-757', u'test_collections-834', u'test_collections-516', u'test_collections-953', u'test_collections-712', u'test_collections-951', u'test_collections-798', u'test_collections-871'] 2021-05-03 06:50:47,614 | test | WARNING | Thread-2772 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 update failed keys from task: [u'test_collections-607', u'test_collections-527', u'test_collections-766', u'test_collections-621', u'test_collections-863', u'test_collections-688', u'test_collections-864', u'test_collections-642', u'test_collections-587', u'test_collections-760', u'test_collections-585', u'test_collections-580', u'test_collections-517', u'test_collections-558', u'test_collections-654', u'test_collections-597', u'test_collections-574'] 2021-05-03 06:50:47,678 | test | WARNING | Thread-2808 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:FHkokJltpxnx-56-574000 delete failed keys from task: [u'test_collections-98', u'test_collections-328', u'test_collections-348', u'test_collections-126', u'test_collections-461', u'test_collections-286', u'test_collections-264', u'test_collections-2', u'test_collections-339', u'test_collections-316', u'test_collections-159', u'test_collections-116', u'test_collections-259', u'test_collections-334', u'test_collections-474', u'test_collections-153', u'test_collections-175', u'test_collections-495', u'test_collections-130'] 2021-05-03 06:50:47,706 | test | WARNING | Thread-2722 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DkhV4g2IJjYDgdV-56-550000 delete failed keys from task: [u'test_collections-73', u'test_collections-358', u'test_collections-311', u'test_collections-278', u'test_collections-364', u'test_collections-321', u'test_collections-266', u'test_collections-281'] 2021-05-03 06:50:47,755 | test | WARNING | Thread-2711 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:a%mMVU_H_3YUxZ3-56-559000 create failed keys from task: [u'test_collections-2828', u'test_collections-2767', u'test_collections-2649', u'test_collections-2681', u'test_collections-2595'] 2021-05-03 06:50:47,789 | test | WARNING | Thread-2773 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 update failed keys from task: [u'test_collections-824', u'test_collections-747', u'test_collections-568', u'test_collections-788', u'test_collections-569', u'test_collections-963', u'test_collections-861', u'test_collections-620', u'test_collections-959', u'test_collections-658', u'test_collections-876', u'test_collections-691'] 2021-05-03 06:50:47,801 | test | WARNING | Thread-2772 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 delete failed keys from task: [u'test_collections-17', u'test_collections-35', u'test_collections-329', u'test_collections-52', u'test_collections-102', u'test_collections-300', u'test_collections-184', u'test_collections-141', u'test_collections-182', u'test_collections-280', u'test_collections-84', u'test_collections-379', u'test_collections-434', u'test_collections-355', u'test_collections-310', u'test_collections-271', u'test_collections-290', u'test_collections-6'] 2021-05-03 06:50:47,871 | test | WARNING | Thread-2881 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:rs--43-291000:kRPGHuE5Bkm_DTs-43-472000 create failed keys from task: [u'test_collections-1928', u'test_collections-2413', u'test_collections-2137', u'test_collections-2315', u'test_collections-1701', u'test_collections-1691', u'test_collections-2583', u'test_collections-2153', u'test_collections-2285'] 2021-05-03 06:50:47,888 | test | WARNING | Thread-2773 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 delete failed keys from task: [u'test_collections-346', u'test_collections-303', u'test_collections-25', u'test_collections-48', u'test_collections-416', u'test_collections-84', u'test_collections-179', u'test_collections-475', u'test_collections-453', u'test_collections-152', u'test_collections-293'] 2021-05-03 06:50:47,973 | test | WARNING | Thread-2772 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:79vO7vaBVKc-56-599000:tSlufAx-Sdhc1UOv_O-56-606000 create failed keys from task: [u'test_collections-2706', u'test_collections-2579', u'test_collections-2747', u'test_collections-2702', u'test_collections-2873', u'test_collections-2752'] 2021-05-03 06:50:48,020 | test | WARNING | Thread-2773 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:79vO7vaBVKc-56-599000:t3gfU0Mgu-Wwt7o-hCjG-56-602000 create failed keys from task: [u'test_collections-2699', u'test_collections-2743'] 2021-05-03 06:50:48,046 | test | WARNING | Thread-2719 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:29IrDQ-U7l1V-56-555000 update failed keys from task: [u'test_collections-687'] 2021-05-03 06:50:48,051 | test | WARNING | Thread-2716 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:f6AaSHeXFEeEmbZbZq%S-56-562000 delete failed keys from task: [u'test_collections-346', u'test_collections-303', u'test_collections-25', u'test_collections-48', u'test_collections-21', u'test_collections-416', u'test_collections-84', u'test_collections-179', u'test_collections-475', u'test_collections-453', u'test_collections-152', u'test_collections-293'] 2021-05-03 06:50:48,053 | test | WARNING | Thread-2720 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Ch%z7dUUqg-LvWf-56-560000 update failed keys from task: [u'test_collections-827', u'test_collections-957', u'test_collections-977', u'test_collections-769', u'test_collections-912', u'test_collections-840', u'test_collections-880'] 2021-05-03 06:50:48,058 | test | WARNING | Thread-2715 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:0-56-559000 update failed keys from task: [u'test_collections-726', u'test_collections-867', u'test_collections-549', u'test_collections-822', u'test_collections-787', u'test_collections-586', u'test_collections-763', u'test_collections-717', u'test_collections-831', u'test_collections-675', u'test_collections-692'] 2021-05-03 06:50:48,059 | test | WARNING | Thread-2723 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:pHRlTFttn3E-4-56-561000 delete failed keys from task: [u'test_collections-424'] 2021-05-03 06:50:48,059 | test | WARNING | Thread-2728 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:vH9-43-755000:eyri-44-105000 create failed keys from task: [u'test_collections-1555', u'test_collections-1665', u'test_collections-1753', u'test_collections-1568', u'test_collections-1634', u'test_collections-1977', u'test_collections-2151', u'test_collections-1630'] 2021-05-03 06:50:48,061 | test | WARNING | Thread-2732 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:oU0RIla5a7M-42-719000:DV-42-931000 create failed keys from task: [u'test_collections-1654', u'test_collections-1723'] 2021-05-03 06:50:48,062 | test | WARNING | Thread-2723 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:pHRlTFttn3E-4-56-561000 create failed keys from task: [u'test_collections-2794'] 2021-05-03 06:50:48,068 | test | WARNING | Thread-2731 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:oU0RIla5a7M-42-719000:Y3rIkI-42-890000 create failed keys from task: [u'test_collections-1933', u'test_collections-1758'] 2021-05-03 06:50:48,069 | test | WARNING | Thread-2729 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:vH9-43-755000:PxCYkH30ichVOO-43-915000 create failed keys from task: [u'test_collections-1309', u'test_collections-1049', u'test_collections-1722', u'test_collections-1142', u'test_collections-1021', u'test_collections-1087', u'test_collections-1286', u'test_collections-1146', u'test_collections-957', u'test_collections-1117', u'test_collections-1316', u'test_collections-1372', u'test_collections-1299', u'test_collections-1432'] 2021-05-03 06:50:48,071 | test | WARNING | Thread-2727 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:gxNNs83S66O-56-552000 update failed keys from task: [u'test_collections-905', u'test_collections-744', u'test_collections-887', u'test_collections-766', u'test_collections-602', u'test_collections-885', u'test_collections-622', u'test_collections-501', u'test_collections-864', u'test_collections-979', u'test_collections-834', u'test_collections-516', u'test_collections-879', u'test_collections-956', u'test_collections-712', u'test_collections-798', u'test_collections-697', u'test_collections-871', u'test_collections-890'] 2021-05-03 06:50:48,105 | test | WARNING | Thread-2716 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:f6AaSHeXFEeEmbZbZq%S-56-562000 create failed keys from task: [u'test_collections-2630'] 2021-05-03 06:50:48,112 | test | WARNING | Thread-2720 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Ch%z7dUUqg-LvWf-56-560000 delete failed keys from task: [u'test_collections-206', u'test_collections-104', u'test_collections-367', u'test_collections-385', u'test_collections-361', u'test_collections-380', u'test_collections-478', u'test_collections-215', u'test_collections-377', u'test_collections-410', u'test_collections-210', u'test_collections-332', u'test_collections-255', u'test_collections-396', u'test_collections-250', u'test_collections-270'] 2021-05-03 06:50:48,128 | test | WARNING | Thread-2715 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:0-56-559000 delete failed keys from task: [u'test_collections-14', u'test_collections-94', u'test_collections-249', u'test_collections-73', u'test_collections-107', u'test_collections-51', u'test_collections-228', u'test_collections-305', u'test_collections-340', u'test_collections-187', u'test_collections-278', u'test_collections-295'] 2021-05-03 06:50:48,165 | test | WARNING | Thread-2715 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:0-56-559000 create failed keys from task: [u'test_collections-2728', u'test_collections-2539'] 2021-05-03 06:50:48,170 | test | WARNING | Thread-2720 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Ch%z7dUUqg-LvWf-56-560000 create failed keys from task: [u'test_collections-2776', u'test_collections-2756', u'test_collections-2872'] 2021-05-03 06:51:04,115 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-05-03 06:53:06,538 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-03 06:53:06,539 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-03 06:53:30,036 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-03 06:53:30,038 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 96% 2021-05-03 06:53:45,654 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-03 06:53:45,654 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-03 06:54:35,937 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:820] data_sets ==> {} 2021-05-03 06:54:53,997 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 11.2309999466 seconds to complete 2021-05-03 06:54:54,016 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2721.132s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_06-09-34/test_1 Logs will be stored at /data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_06-09-34/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,rerun=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops', 'quota_percent': '95', 'ini': '/tmp/win10-bucket-ops.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even/logs/testrunner-21-May-03_06-09-34/test_2', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 2, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-03 06:55:05,272 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 06:55:06,489 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 06:55:07,585 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d55c type=client open_count=1 channel=[id: 0x3043b8b7, 0.0.0.0/0.0.0.0:54705] timeout=300.0>) 2021-05-03 06:55:13,963 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d55d type=client open_count=1 channel=[id: 0x0478ed17, 0.0.0.0/0.0.0.0:49389] timeout=300.0>) 2021-05-03 06:55:16,019 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d55e type=client open_count=1 channel=[id: 0x56e252a3, 0.0.0.0/0.0.0.0:36507] timeout=300.0>) 2021-05-03 06:55:18,049 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d55f type=client open_count=1 channel=[id: 0xe6ca0e53, 0.0.0.0/0.0.0.0:58675] timeout=300.0>) 2021-05-03 06:55:20,079 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d560 type=client open_count=1 channel=[id: 0x88e6f028, 0.0.0.0/0.0.0.0:38323] timeout=300.0>) 2021-05-03 06:55:22,105 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d561 type=client open_count=1 channel=[id: 0x32e9bbf9, 0.0.0.0/0.0.0.0:42953] timeout=300.0>) 2021-05-03 06:55:24,128 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d562 type=client open_count=1 channel=[id: 0x87a422de, 0.0.0.0/0.0.0.0:46725] timeout=300.0>) 2021-05-03 06:55:26,148 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d563 type=client open_count=1 channel=[id: 0xf07f255a, 0.0.0.0/0.0.0.0:33477] timeout=300.0>) 2021-05-03 06:55:28,174 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.196:8091 during connect (<_realsocket at 0x5d564 type=client open_count=1 channel=[id: 0xc2faf6a8, 0.0.0.0/0.0.0.0:59554] timeout=300.0>) 2021-05-03 06:55:30,196 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.196:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d565 type=client open_count=1 channel=[id: 0x4deca8d0, 0.0.0.0/0.0.0.0:33065] timeout=300.0>) 2021-05-03 06:55:38,548 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d566 type=client open_count=1 channel=[id: 0x417f5690, 0.0.0.0/0.0.0.0:45501] timeout=300.0>) 2021-05-03 06:55:40,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d567 type=client open_count=1 channel=[id: 0xe2bc0be0, 0.0.0.0/0.0.0.0:54199] timeout=300.0>) 2021-05-03 06:55:42,609 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d568 type=client open_count=1 channel=[id: 0x24538a66, 0.0.0.0/0.0.0.0:41048] timeout=300.0>) 2021-05-03 06:55:44,632 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d569 type=client open_count=1 channel=[id: 0x768ab7af, 0.0.0.0/0.0.0.0:42689] timeout=300.0>) 2021-05-03 06:55:46,683 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d56a type=client open_count=1 channel=[id: 0xebb1a994, 0.0.0.0/0.0.0.0:32967] timeout=300.0>) 2021-05-03 06:55:48,706 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d56b type=client open_count=1 channel=[id: 0xee0fb425, 0.0.0.0/0.0.0.0:48911] timeout=300.0>) 2021-05-03 06:55:50,733 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d56c type=client open_count=1 channel=[id: 0x183c72ce, 0.0.0.0/0.0.0.0:49295] timeout=300.0>) 2021-05-03 06:55:52,757 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.98.195:8091 during connect (<_realsocket at 0x5d56d type=client open_count=1 channel=[id: 0x145bc28c, 0.0.0.0/0.0.0.0:42145] timeout=300.0>) 2021-05-03 06:55:54,786 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.98.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d56e type=client open_count=1 channel=[id: 0x07789f46, 0.0.0.0/0.0.0.0:43547] timeout=300.0>) 2021-05-03 06:56:02,703 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d56f type=client open_count=1 channel=[id: 0x07dba01c, 0.0.0.0/0.0.0.0:58701] timeout=300.0>) 2021-05-03 06:56:04,726 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d570 type=client open_count=1 channel=[id: 0x8375c9fd, 0.0.0.0/0.0.0.0:41241] timeout=300.0>) 2021-05-03 06:56:06,785 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d571 type=client open_count=1 channel=[id: 0x7cd9e871, 0.0.0.0/0.0.0.0:45901] timeout=300.0>) 2021-05-03 06:56:08,842 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d572 type=client open_count=1 channel=[id: 0xb5aeb947, 0.0.0.0/0.0.0.0:34095] timeout=300.0>) 2021-05-03 06:56:10,864 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d573 type=client open_count=1 channel=[id: 0x8cc9a232, 0.0.0.0/0.0.0.0:59622] timeout=300.0>) 2021-05-03 06:56:12,887 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d574 type=client open_count=1 channel=[id: 0xa91741dc, 0.0.0.0/0.0.0.0:38341] timeout=300.0>) 2021-05-03 06:56:14,908 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d575 type=client open_count=1 channel=[id: 0x10e5bf9c, 0.0.0.0/0.0.0.0:46194] timeout=300.0>) 2021-05-03 06:56:16,944 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d576 type=client open_count=1 channel=[id: 0x37189534, 0.0.0.0/0.0.0.0:37107] timeout=300.0>) 2021-05-03 06:56:18,969 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.121.10:8091 during connect (<_realsocket at 0x5d577 type=client open_count=1 channel=[id: 0x1b43c043, 0.0.0.0/0.0.0.0:57627] timeout=300.0>) 2021-05-03 06:56:20,993 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.121.10:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d578 type=client open_count=1 channel=[id: 0x2c8354cc, 0.0.0.0/0.0.0.0:51585] timeout=300.0>) 2021-05-03 06:56:29,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d579 type=client open_count=1 channel=[id: 0xdf4ab3c8, 0.0.0.0/0.0.0.0:48386] timeout=300.0>) 2021-05-03 06:56:31,686 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d57a type=client open_count=1 channel=[id: 0xfe5e8b44, 0.0.0.0/0.0.0.0:36963] timeout=300.0>) 2021-05-03 06:56:33,721 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d57b type=client open_count=1 channel=[id: 0x83bf16e1, 0.0.0.0/0.0.0.0:43947] timeout=300.0>) 2021-05-03 06:56:35,778 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d57c type=client open_count=1 channel=[id: 0xca41849f, 0.0.0.0/0.0.0.0:55126] timeout=300.0>) 2021-05-03 06:56:37,803 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d57d type=client open_count=1 channel=[id: 0x24579db6, 0.0.0.0/0.0.0.0:34579] timeout=300.0>) 2021-05-03 06:56:39,825 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d57e type=client open_count=1 channel=[id: 0x8b089284, 0.0.0.0/0.0.0.0:39461] timeout=300.0>) 2021-05-03 06:56:41,846 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d57f type=client open_count=1 channel=[id: 0x3ecfd271, 0.0.0.0/0.0.0.0:60951] timeout=300.0>) 2021-05-03 06:56:43,880 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.186:8091 during connect (<_realsocket at 0x5d580 type=client open_count=1 channel=[id: 0x2dae86f2, 0.0.0.0/0.0.0.0:42661] timeout=300.0>) 2021-05-03 06:56:45,917 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.186:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d581 type=client open_count=1 channel=[id: 0xc54d621a, 0.0.0.0/0.0.0.0:56614] timeout=300.0>) 2021-05-03 06:56:51,203 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d582 type=client open_count=1 channel=[id: 0xa3977420, 0.0.0.0/0.0.0.0:53739] timeout=300.0>) 2021-05-03 06:56:53,229 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d583 type=client open_count=1 channel=[id: 0xb762cc7d, 0.0.0.0/0.0.0.0:54855] timeout=300.0>) 2021-05-03 06:56:55,262 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d584 type=client open_count=1 channel=[id: 0xe4a03b87, 0.0.0.0/0.0.0.0:59535] timeout=300.0>) 2021-05-03 06:56:57,299 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d585 type=client open_count=1 channel=[id: 0xe8721d18, 0.0.0.0/0.0.0.0:33919] timeout=300.0>) 2021-05-03 06:56:59,319 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.120.206:8091 during connect (<_realsocket at 0x5d586 type=client open_count=1 channel=[id: 0x327573c3, 0.0.0.0/0.0.0.0:59201] timeout=300.0>) 2021-05-03 06:57:01,349 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.120.206:8091/nodes/self error [Errno 111] Connection refused 2021-05-03 06:57:19,243 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:595] RAM quota was defined less than 100 MB: 2021-05-03 06:57:34,377 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.98.196, 172.23.98.195, 172.23.121.10, 172.23.104.186, 172.23.120.206 initialized 2021-05-03 06:57:34,433 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 06:57:34,503 | test | INFO | MainThread | [basetestcase:log_setup_status:544] ========= CollectionBase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-05-03 06:58:13,526 | test | INFO | pool-4-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.98.196 | kv | 7.0.0-5085-enterprise | 10.3909205549 | Cluster node | | 172.23.98.195 | None | | | <--- IN --- | | 172.23.121.10 | None | | | <--- IN --- | | 172.23.104.186 | None | | | <--- IN --- | | 172.23.120.206 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-03 06:58:13,555 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-03 06:58:23,678 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-03 06:58:23,759 | test | INFO | pool-4-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2319998741 sec 2021-05-03 06:58:24,385 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-03 06:59:02,516 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-03 06:59:02,914 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 483167656 | 177216428 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 486370936 | 126980063 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 521559520 | 144546460 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-03 06:59:02,914 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-03 07:00:28,963 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-05-03 07:00:28,974 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations Build was aborted Aborted by Balakumaran G Archiving artifacts